XWiki AWS Distributions

Development
, ,
1

Hi devs,

Context

Sanchita and Sachin have worked for the GSOC 2021to support deploying XWiki on AWS technology. There are basically 3 solutions that were developed (see GSOC 2021- Sanchita Singh - Work Summaries - #7 by sanchita):

  • Using CDK
  • Using CF Template
  • Using an AMI

Using CDK

See Cdk xwiki.mp4 - Google Drive

This method just requires that you git clone an XWiki repo, install some amazon tools locally and have an AWS account.

Using CF Template

See Xwiki cloudformation 2.mp4 - Google Drive

This method requires that you have an AWS account and use the AWS UI to configure an XWiki setup. It also requires a CF Template (a json file) which we can provide through some download link or by uploading one into a S3 bucket (this needs an AWS account to hold it, and pay for it).

Using an AMI

This allows to publish the AMI into the AWS Marketplace. It requires to build such an AMI every time XWiki is released and right now the generated XWiki setup is not production-ready (it’s more for demos). For users they just need an AWS account to use it. To publish it you need an AWS account.

Issues

It’s not possible for xwiki.org to have an AWS account since that requires a bank account. Thus it’s not possible for xwiki.org to publish the CF Template into an S3 bucket and to publish the AMI in the marketplace.

Proposal 1

The xwiki.org open source project focuses on providing the following:

  • the git repo/source for the CDK way
  • be able to generate the CF Template but don’t publish it into an S3 bucket. Users can use it by uploading it into the AWS UI
  • be able to generate the AMI but don’t publish it to the marketplace

Then, sponsoring companies (like XWiki SAS) can take over and do the remaining work if they’re interested:

  • publish the CF Template into an S3 bucket using their own AWS account
  • publish the AMI into the marketplace using their own AWS account

Pros:

  • Clear separation of concern between xwiki.org and XWiki SAS (or other companies/individuals wanting to publish XWiki on AWS)
  • Companies can also add a fee on the AMI and provide more services (support, etc).
  • The AMI (and S3 bucket for the CF Template) would be advertised in the sponsoring company zones, including in the download area. For ex, right now XWiki SAS proposes XWiki Cloud there. It could propose AMI/CFT S3 bucket too, in the same area.

Cons:

  • Not 100% consistent with other distributions that the xwiki.org project publishes like the docker image (since the project wouldn’t publish the AMI for example).

Proposal 2

Create a XWiki.org AWS account but use the bank account information from XWiki SAS which would just act as a money holder. There’s one potential issue: XWiki SAS is interested to make the AMI paying (at a small fee to show that open source is not free) and give back the money to the open source project (not so easy to implement in a contractual way but seen the sponsoring that XWiki SAS already does for the open source project, I believe it has enough credibility to be trusted by the community!).

Pros:

  • xwiki.org handles all distributions it supports (including the AWS one)
  • Slightly better for active installs IMO since the AWS distribution would be an official one for xwiki.org. Note that this is not completely true since with proposal 1, it’s also official but only with the CDK and CFT mechanisms (without S3). There’s only the AMI option not official and provided by XWiki SAS.

Cons:

  • The AWS account would be managed both by xwiki.org community and XWiki SAS which is not very clean. E.g. imagine that XWiki SAS wants to add, say 2 euros to the AMI price (over the AWS price for usage). How is this decided by the community along with XWiki SAS?

Proposal 3

Decide that xwiki.org doesn’t want to propose new official distributions (the AWS ones) to reduce maintenance costs.

Pros:

  • Less maintenance, support, and documentation work

Cons:

  • Would not get us more active installs

Conclusion

I’m hesitating between 1 and 2. Yesterday, I was finding proposal 1 to be the best by far. Today, I’m hesitating with proposal 2 which has some benefits but I don’t find the paying option easy to handle in a community way. So I think that globally I’m still leaning towards proposal 1.

WDYT?

Thanks

1 Like
2

Imho, Proposal 1 seems to be the correct approach. XWiki.org is an open source org/project. The only reason why we were uploading the CF template to s3 bucket was for ease of installation for end users since they can directly go to a URL and everything will be prefilled . If we can go via download the CF template from Github and upload to AWS UI then we can totally move away from managing an AWS account that requires a bank account. This decision is backed by the fact that the AMI design doesn’t take into account single responsibility model for development. Everything is currently being managed by one single EC2 instance whereas the CDK and Cloudformation template makes use of wide range of AWS Services to provide for a highly scalable and available solution for XWiki production deployments. If XWiki SAS wants to charge a small amount of fee using the AMIs , then they can publich it in their own AWS account alongside providing support for “no-download-template-and-upload” of CF templates.

1 Like
3

Not to mention, merging XWiki org with SAS for AWS Distributions and adding fee on top of it (how minimal it might be) might also affect the way how future GSoC aspirants perceive the org.

1 Like
4

I agree that Proposal 1 seems the most logical approach here. I’m not really sure it would have a bad impact on active installs if instructions to use AWS are clear enough and if XWiki SAS provides a very easy setup for AWS (even with small fees).

I don’t like much the idea of mixing XWiki SAS and .org just for having XWiki.org in the AWS marketplace. I guess you put proposal 3 to be complete, I don’t know how much maintenance it would be to maintain the files for the AWS distribution, but I don’t expect it to be a lot of maintenance. So at the very least I think it worths it to try it for a year or two, and see if we have people using it.

So to sum up I’d be +1 for approch 1, -1 for approach 2 and -1 for approach 3.

1 Like
5

I have the same feeling that Proposal 1 (with clear separation between xwiki.org and XWiki SAS) is the right approach. +1 for it.

Thanks,
Marius

1 Like
6

First of all, congrats to Sanchita for the work and to Sachin for his guiding! The videos are really helpful at visualizing the result.

Re the options, AFAIU, the only real issue is about the publishing of the AMI in the marketplace and who does it an under which terms.

The whole S3 bucket storage convenience options seems less interesting to me, as I noticed in the video you can simply upload the configuration that you can get from an XWiki official GitHub repository. Actually, instead of an S3 URL, can’t you just specify the raw GitHub URL to the needed json file? Maybe I’m missing something here.

Under ideal conditions, I think I would favor a Proposal 4 (which is basically a mix of 1 and 2 that still keeps things separate, IMO), which would be to:

  • Publish a “free” and “official” AMI under the xwiki.org name (even if for formality sake, the bank account details would correspond to XWiki SAS). “free” = 0 cost on top of the AWS operation costs. This AMI would provide no support and would be the equivalent of the “community” support that you get when you download XWiki for yourself. Of course, this publishing would need to be part of the release process, as mentioned, so that the “free” offering is always kept up to date and effortless for anyone to try the latest XWiki version on an AWS AMI.
  • Allow anyone/companies to publish additional AMIs into the marketplace using their own AWS account. This would, of course, include support from the publishing company for the extra fee they add to their offering, on top of the AWS operations fee. The 3rd party supported/published XWiki versions will not always be in synch with the latest/official/free/xwiki.org AMI published offering, but that’s fine. In practice, it would probably be XWiki SAS. Not sure why it needs to be a sponsoring company or how/why we could/would restrict that.

IMO, this approach would be free from any org vs SAS problems, while staying true to the spirit of what we have done so far.

Otherwise, if only Proposals 1-3 are on the table, I’m obviously in favor of Proposal 1, and leave 3rd parties in charge of publishing XWiki as AWS AMI. It’s not ideal, as companies might lag in updating the XWiki version, for various reasons, but we would have to live with that.

2 Likes
7

Thankyou @Enygma.

Actually, it won’t work with the Github link of the template as you need to specify the URL of a public S3 bucket or manually upload the template.

8

Yes, Sachin also explained it to me and I now understand it as, most likely, an Amazon business decisions to force that URL to be S3 only. Anyway, I still don’t see a problem in instructing someone to download a json from an XWiki maintained GitHub repository and upload it to AWS.

9

For that @vmassol suggested to rather upload a copy of the template on the documentation page only. I updated the documentation and uploaded a modified video according to proposal 1 in the same drive folder

10

IMO, having that CF template (which is basically, infrastructure as code) attached as a file on a wiki document is a very bad practice.

I strongly suggest that we treat it like code and have proper version control (and many other github provided features) for it inside a git repository that we can reference (either directly the file or the repo itself) in the documentation. I believe that the best fit would be a new and dedicated (e.g. aws-cf) contrib repository instead of bundling it in the existing aws repo, which we could even rename to aws-cdk instead of bundling too many distributions in one and making harder the process of reporting an issue for one of the distribution methods.

1 Like
11

We already have separate repositories for both methods. (GitHub - xwiki-contrib/aws: Different packagings to deploy XWiki on AWS and GitHub - xwiki-contrib/aws-xwiki-cloudformation). And yes, what you are suggesting is an option. that will require just a few extra steps for installation.

1 Like
12

Yes. As I mentioned to @sanchita in the past, we need to have releases and git tags with versions.

13

I’m currently dropping this thread as I believe that for this to work, we would first need to have a champion for AWS in the active committers, and I don’t see one ATM.

We can revive this if we get such a champion in the future. It could also be the topic of a future GSOC to improve on what was built (but we’d need a committer mentoring it and taking over at the end).

Thanks