New Filter Streams Extension: GitHub Importer

Development
, ,
#42

Hi @vmassol, the LaTeX extension’s tests are failing on my end by doing mvn clean install or mvn test. What I did:

  1. git clone https://github.com/xwiki-contrib/latex
  2. mvn test or mvn clean install

So I am following xwiki-platform-mail which has no test failures on my end and as mentioned on the DockerTesting page in Full examples.
About my previous post: I was asking how to run the tests, what do I need to do with docker after installing locally and how to have the Run Test button like you are having on the youtube video. I know I am lacking some configurations that’s why I asked since I have no idea what I am missing and the page https://dev.xwiki.org/xwiki/bin/view/Community/Testing/DockerTesting doesn’t mention about it. There’s also a lot of information on that page which mixes up a lot of things and creates confusion. (Edit1) And there’s also no FLV video files in the target directory when I do mvn test on xwiki-platform-mail.
So today, I was learning the xwiki-platform-mail tests and I was running the tests by mvn test but it would be better if I can test inside IntelliJ IDEA. (Edit2) It’s not even recognizing what @UITest is.
Let me know if I am missing something or am unclear about my post.

#43

More info on the LaTeX extension’s test failures:

Tests run: 79, Failures: 37, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for LaTeX - Parent POM 1.10-SNAPSHOT:
[INFO]
[INFO] LaTeX - Parent POM ................................. SUCCESS [  6.829 s]
[INFO] LaTeX - Syntax ..................................... FAILURE [ 59.663 s]
[INFO] LaTeX - Filter ..................................... SKIPPED
[INFO] LaTeX - Export ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:16 min
[INFO] Finished at: 2020-06-18T01:00:50+05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project latex-syntax: There are test failures.
[ERROR]

File in ./target/surefire-reports: JUnit Vintage.txt (279.2 KB)

#44

Today, I will try to write the tests for the GitHub Importer by following xwiki-platform-mail, xwiki-platform-menu and latex.

#45

Summary of what’s done so far for the first release:

  1. GHIMPORT-1
  2. GHIMPORT-2
  3. GHIMPORT-6
  4. Continuous Integration
  5. Created Extension Pages:

The tests are done manually, the code tests will be created soon.
I hope everything is going fine so far, if not, please let me know.
Posted a Request for Nexus account.

Thanks!

#46

Extension released: GitHub Importer 1.0 version.
GitHub Importer Filter 1.0 jar on nexus
Feel free to reopen the issues: https://jira.xwiki.org/projects/GHIMPORT/issues/GHIMPORT-2?filter=doneissues
E.X.O is down atm (503 Service Unavailable) so I will modify the pages tomorrow (for download and release notes).
Edit 1: Checked again after few minutes, it’s up. Continuing.
Edit 2: Not working again. Really tired now, will continue in the morning.
Edit 3: I am aware that my other extension pages are incomplete / wrong.
Next is to do a forum post in News, which will be done once the extension is available for download.
Edit 4: Fixed extension pages, posted a new extension forum in news.

Any problems, let me know. Thanks!

#47

Update: Created an issue on JIRA for Simplified UI whose progress will start after second release (Roadmap 3).

Roadmap 2:

Issues to be resolved:

Requirements to be completed:

  • 4
#48

So far, I am able to run docker tests with videos created in target/…/screenshots folder. But have not been able to configure IntelliJ project with docker. Although I’ve installed docker plugin and used the services as suggested by jetbrains. But I don’t think this is even required since I’ve read the DockerTesting page. So either the dependencies are missing or not imported or IntelliJ is not recognizing the test file.
e.g. @UITest notation is not recognized by IntelliJ IDEA.
I strongly believe this is due to dependencies missing. And I am not talking about this project, Latex extension has similar problems (which is also a contrib project).
e.g. ViewPage at Latex extension’s ExportIT test is not recognized in the project IDE. Even the @UITest notation as well. I have been trying to make tests without this by mimicking these tests but it’s hard since the IDE is not recognizing the types etc.
Some one who wants to test this can try to reproduce by:

  1. Deleting (or rename) ~\.m2\repository\org\xwiki.
  2. Then clone Latex.
  3. Open parent pom (pom.xml) of latex in IntelliJ IDEA in the root directory of latex extension just cloned.
  4. Let IntelliJ IDEA resolve dependencies. (make sure you have configured settings.xml in m2, see Building)
  5. Open ExportIT.java file and check if IntelliJ recognizes anything.
  6. If issue is not reproduced, let me know here or on chat!

I am not blocked by this. Just posting all this here to let forum watchers know I am not wasting time but spending it on this to work so in future it will solve many problems.
I am not sure if it’s a bug of XWiki, but possibly the extension’s pom.

Edit: Fixed by activating integration-tests and docker in maven profiles!

#49

This is not correct as it doesn’t match wht’s in JIRA: https://jira.xwiki.org/projects/GHIMPORT/versions/18572

Also missing GHIMPORT-7 and missing a jira about the docker tests.

Thanks!

#50

Fixed :slight_smile:
Let me know if still incorrect.

#51

Sounds good. So, to recap:

Thanks

1 Like
#52

Hi @vmassol . I wanted to let you know all news about university which has been released a day ago.
So far what’s been done:

  1. Syntax conversion
  2. Basic authentication

I am still working on the UI and docker tests (will be re-written for the newly created UI instead of the Filter Application UI which exist at the moment making the build unstable).
So, about my studies:

  1. 5th July - Final Year Project Evaluation (1st evaluation, then final coming soon) [completed].
  2. 6th July - Mobile App development Lab final.
  3. Final exams are starting from 13th July ending 17th July.
  4. Final Year Project presentation and final evaluation is on 20th July (The date might change, will post here if it does).
  5. Most of the quizzes and assignments have passed in June, few remaining in July upto 10th July.

So that’s all, a busy month of July for studies. But it’s good that I will be free and can focus on GitHub Importer after 20th or 25th (will post here when the final semester is over).
I will try to finish what I can for the 6th July release of GitHub Importer and will let you know or will ask your help on what we can do if everything is not going as intended.

Thank you!

#53

Ok, so I just commited. Build is stable, removed the unstable docker tests.
Now remains:

  • Docker Testing

I have replaced the Sort pages issue by Syntax conversion, will include that in the next release (I hope you are okay with that).
And will try to create docker test after the Lab final.
See you tomorrow :slight_smile:

#54

Hi @haxsen. Do you mean “integration tests” instead of “docker testing”? https://github.com/xwiki-contrib/application-github-importer/tree/master/github-importer-filter is still missing a test directory for integration tests for the filter (see my previous PR review where I mentioned it several times but since you didn’t ack it, I don’t know if you understood - I even pointed you to an example on how to write it). Let me know. Thx

#55

Hi @vmassol I meant docker testing since we now have the GitHub Importer UI.
Ok, I think there is a desync in our communication.
This is how I understood everything:

  1. You mentioned about docker tests in the very beginning, pointed me to https://dev.xwiki.org/xwiki/bin/view/Community/Testing/DockerTesting .
  2. I created Docker test (functional) using Filter Streams UI (which is how user interacted with the GitHub Importer in the release 1). But it had errors.
  3. So you mentioned 3 things then:
    1. GitHub Importer needs UI.
    2. Docker tests (functional) will be created using GitHub Importer UI.
    3. Unit tests execute faster.
  4. So then I created the UI.
  5. My next step was to create Docker Tests (since that’s how I understood).

Now this has confused me, honestly.

The release 2 contained following issues to be resolved:

  1. Sort pages (shifted to release 3, replaced by Syntax conversion).
  2. Basic authentication.
  3. User Interface (GitHub Importer UI).
  4. Docker Tests (func tests).

But since you are talking about Unit Tests, I will create the Unit Tests.

So far what’s done for release 2:

  1. Basic Authentication.
  2. Syntax Conversion.
  3. UI

So my next step is:

  • Unit Tests.

Is this fine?
Please clear my confusions and let me know which issues need to be resolved, other than what’s already done. (and also if I should create an issue on JIRA for Unit tests to track?).

Thanks.

#56

To be precise: A Custom UI. It already has a UI through the Filter Streams App but it’s a generic UI that is not easy to use.

These are not done yet? Especially 1, 2 and 3?

Can you please provide a status for each of them?

I’m not asking anything. I’m providing guidance to you. You are the one responsible for the implementation of this extension. So you shouldn’t do something because someone says so, but because you believe in it. If you don’t, then you need to raise it ofc.

Right now you don’t have any tests verifying that the filter works. That’s not good. You started writing them using the docker test framework (at my suggestion) but you didn’t realize that in your tests you were testing the Filter Stream App UI and there’s no point in doing that. I noticed this and pointed you to writing integration tests for the filter, which are much easier to write and execute. I also said that you’ll also need docker tests for the custom UI when you code it.

In term of priority, it seems to me that it’s better to test that the code you have already written rather than focusing on writing tests (docker tests) for something that is not finished yet (the custom UI). Is it?

In practice you need tests for both but you’re lagging behind so if there can only be one for the release, it’s the integration tests.

Today is supposed to be release day and you have 4 issues opened out of 4 in https://jira.xwiki.org/projects/GHIMPORT/versions/18572 This is 0% achievement.

I hope it’s just that you haven’t updated the issues. Let us know.

I already answered this one in the past. See https://dev.xwiki.org/xwiki/bin/view/Community/DevelopmentPractices#HRule:Don27tcreateunnecessaryissues

So the answer is that you don’t write a jira issue to remember to code an issue or to remember to document one or to remember to follow best practices, right? :slight_smile: It’s the same for tests, they’e not optional. They’re part of coding and you should normally not push something without a test to prove that it works. Right now you’ve created technical debt since you didn’t have that. Let’s fix it quickly.

Now if you want an issue to help yourself to remember it, go ahead. Make sure it has a ira component of “development issue only” in this case since users don’t care about this.

Thanks

#57

Thanks for clearing the confusion.
The 3 issues are resolved, haven’t updated since they’re missing their tests. Manually tested them and they work fine. I need to do Unit tests first and then functional tests, now that I know.

I see, I should complete the tests to prove that they work.

So, only 1 issue remains for this release, i.e: Integration Tests.
Today I will try to finish this and get ready for the release :slight_smile:
If I don’t finish, I may ask for release date extension, but I hope that won’t occur.

Thanks!

#58

Hi @vmassol , I wanted to release the extension on 6th july but CI was stuck on:

 [Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

You can check here: https://ci.xwiki.org/view/Contrib/job/XWiki%20Contrib/job/application-github-importer/job/master/10/console
I waited for half an hour but it was still stuck, searching online revealed it might be out of space. But I think maybe it was caused by my last 2 commits which were made in 2 minute intervals. I am not sure what’s wrong.
But I will continue the release in the morning, it’s sad but I tried my best to release on 6th july.
After the release, I will write an extension page for GitHub Importer (with its own UI instead of Filter UI). Should GitHub Importer Filter 's extension page be deleted (since it’s overtaken by custom UI)?

Thanks.

#59

It’s not stuck. It’s waiting for an agent to become available.

It did build but your tests are failing. They should also be failing for you locally normally. See https://ci.xwiki.org/view/Contrib/job/XWiki%20Contrib/job/application-github-importer/job/master/11/#showFailuresLink

#60

No it’s fine to keep both, they are different modules. Each needs to have the documentation that matches what it contains. so the UI one will document the new UI and the filter one is good as it is now; it’ll just need an info box to explain that there’s a better and simpler UI available to use and point to the new UI extension.

1 Like
#61

I pushed from Linux, I think that is creating the problem.
The tests are passing on Linux but failing on Windows with same errors as Jenkins.
And it is also adding unneeded characters in the test when I test on windows.

Is there an option to change Jenkins agent to be compatible with Linux instead of Windows?

We need to set it to Linux since there are also functional tests coming in the next releases and they do not work on windows (we discussed the errors of XWiki environment not getting set on Windows but worked on Linux).

Or is there a better solution?