Functional tests fail in xwiki-platform 10.8

Help / Discuss
1

Dear all,

in the xwiki-platform developer guide https://dev.xwiki.org/xwiki/bin/view/Community/Testing#HFunctionalTesting, it said that there are two locations for the functional tests:

"
_in the past we were **_putting functional tests in xwiki-platform-distribution/xwiki-platform-distribution-flavor/xwiki-platform-distribution-flavor-test_** _

_but we have started to move them inside the specific modules they are testing. _
Name the tests using IT.java
"

I focused on running function tests. I have done the following:
Case 1) I runned the functional test in distribution-flavor-test , almost all functional tests passed.
Case 2) I runned the functional test in *specific modules , and the tests are in IT.java.
half of the tests failed.

My question is:

    1. the above Case 2) is normal or not?
      Why a lot of the case 2) testcases fail?
    1. In 10.8 version, I found the number of tests in Case 1) are largely more than that in Case 2). I want to confirm my observation is right or not?
  • 3.Do the above two cases cover all the functional tests? if not? where else can i find the other functional tests?

Thanks.
Best regards,
Wuxia

2

Nothing should fail except possible known flickering tests which can sometimes fail and that we need to fix. They’ logged here: Loading...

No that shouldn’t be the case. Can you provide an example of a test in 10.8 but not in master? Also WDYM by “largely”? 1 more? 10 more? 100 more? :slight_smile:

That should cover all. Could you tell us why you’re interested in locating functional tests? I’m curious!

Thanks

3

Hi,

I am a student, doing research about using software analysis methods to analyze software projects. ’
In my experiments, I use XWKI-platform 10.8 as the subject.
I am interested in the code functional test, because I want to get tracing data under the functional tests.
To get correct and complete data, I need make functional test run successfully, and also locate all functional tests as possible.

In xwiki10.8 version:

In above Case 1), I locate functional tests in 7 modules. My test results in each module are:

    1.xwiki-platform-distribution-flavor-test-escaping
      run=652， skipped=104.

    2.xwiki-platform-distribution-flavor-test-jmeter   
     perfromance test (I ignore this module )

    3.xwiki-platform-distribution-flavor-test-misc         
      run 4

    4.xwiki-platform-distribution-flavor-test-selenium      
      run = 50, failure = 6, errors = 2.

    5.xwiki-platform-distribution-flavor-test-storage 
      run=11, skipped=1

    6.xwiki-platform-distribution-flavor-test-ui                 
      run=148,  failure=2, errors=6, skipped=1

    7.xwiki-platform-distribution-flavor-test-upgrade         
      run 1, error 1

in above Case 2), I locate functional tests in 7 modules. my test results for each module are:

    1. xwiki-platform-activeinstalls/xwiki-platform-activeinstalls-test/xwiki-platform-activeinstalls-test-tests
      run 1, failure 1.

    2. xwiki-platform-administration/xwiki-platform-administration-test/xwiki-platform-administration-test-tests    
        run 11, failure 1,  errors 4.

    3. xwiki-platform-extension/xwiki-platform-extension-test/xwiki-platform-extension-test-tests
        run 15, failures 0, errors 0

    4. xwiki-platform-filter/xwiki-platform-filter-test/xwiki-platform-filter-test-tests
        run 1, failure 1.

    5. xwiki-platform-menu/xwiki-platform-menu-test/xwiki-platform-menu-test-tests
        run 1, failure 1

    6. xwiki-platform-notifications/xwiki-platform-notifications-test/xwiki-platform-notifications-test-tests
      run 8,  failure 2, errors 1

    7. xwiki-platform-scheduler/xwiki-platform-scheduler-test/xwiki-platform-scheduler-test-tests
      run 1, failure 1

In summary, I found the number of case1 ) testcases are larger than the number of case 2) test cases.
Also, case2) test cases fail a lot when compared to case 1).

Since case 1) covers a lot of xwiki-platform functions (large number of testcases),
I am wondering if I can ignore the case2), that is, only use the data coolected in case 1) for my experiments?

Thanks for your reply.
Best regards,
Wuxia

4

Sounds cool. Could you give you more info on what you’re researching? And thanks for choosing XWiki

5

There are more than 7 modules having functional tests in xwiki-platform-core:

vmassol ~/dev/xwiki/xwiki-platform (master *+) $ find . -name "*test-tests" -type d
./xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-test/xwiki-platform-annotation-test-tests
./xwiki-platform-core/xwiki-platform-filter/xwiki-platform-filter-test/xwiki-platform-filter-test-tests
./xwiki-platform-core/xwiki-platform-activeinstalls/xwiki-platform-activeinstalls-test/xwiki-platform-activeinstalls-test-tests
./xwiki-platform-core/xwiki-platform-scheduler/xwiki-platform-scheduler-test/xwiki-platform-scheduler-test-tests
./xwiki-platform-core/xwiki-platform-rest/xwiki-platform-rest-test/xwiki-platform-rest-test-tests
./xwiki-platform-core/xwiki-platform-administration/xwiki-platform-administration-test/xwiki-platform-administration-test-tests
./xwiki-platform-core/xwiki-platform-index/xwiki-platform-index-test/xwiki-platform-index-test-tests
./xwiki-platform-core/xwiki-platform-webjars/xwiki-platform-webjars-test/xwiki-platform-webjars-test-tests
./xwiki-platform-core/xwiki-platform-notifications/xwiki-platform-notifications-test/xwiki-platform-notifications-test-tests
./xwiki-platform-core/xwiki-platform-extension/xwiki-platform-extension-test/xwiki-platform-extension-test-tests
./xwiki-platform-core/xwiki-platform-observation/xwiki-platform-observation-test/xwiki-platform-observation-test-tests
./xwiki-platform-core/xwiki-platform-webdav/xwiki-platform-webdav-test/xwiki-platform-webdav-test-tests
./xwiki-platform-core/xwiki-platform-mail/xwiki-platform-mail-test/xwiki-platform-mail-test-tests
./xwiki-platform-core/xwiki-platform-resource/xwiki-platform-resource-test/xwiki-platform-resource-test-tests
./xwiki-platform-core/xwiki-platform-xclass/xwiki-platform-xclass-test/xwiki-platform-xclass-test-tests
./xwiki-platform-core/xwiki-platform-repository/xwiki-platform-repository-test/xwiki-platform-repository-test-tests
./xwiki-platform-core/xwiki-platform-panels/xwiki-platform-panels-test/xwiki-platform-panels-test-tests
./xwiki-platform-core/xwiki-platform-user/xwiki-platform-user-directory/xwiki-platform-user-directory-test/xwiki-platform-user-directory-test-tests
./xwiki-platform-core/xwiki-platform-user/xwiki-platform-user-profile/xwiki-platform-user-profile-test/xwiki-platform-user-profile-test-tests
./xwiki-platform-core/xwiki-platform-wiki/xwiki-platform-wiki-test/xwiki-platform-wiki-test-tests
./xwiki-platform-core/xwiki-platform-watchlist/xwiki-platform-watchlist-test/xwiki-platform-watchlist-test-tests
./xwiki-platform-core/xwiki-platform-sharepage/xwiki-platform-sharepage-test/xwiki-platform-sharepage-test-tests
./xwiki-platform-core/xwiki-platform-blog/xwiki-platform-blog-test/xwiki-platform-blog-test-tests
./xwiki-platform-core/xwiki-platform-flamingo/xwiki-platform-flamingo-skin/xwiki-platform-flamingo-skin-test/xwiki-platform-flamingo-skin-test-tests
./xwiki-platform-core/xwiki-platform-flamingo/xwiki-platform-flamingo-themes/xwiki-platform-flamingo-theme-test/xwiki-platform-flamingo-theme-test-tests
./xwiki-platform-core/xwiki-platform-release/xwiki-platform-release-test/xwiki-platform-release-test-tests
./xwiki-platform-core/xwiki-platform-linkchecker/xwiki-platform-linkchecker-test/xwiki-platform-linkchecker-test-tests
./xwiki-platform-core/xwiki-platform-messagestream/xwiki-platform-messagestream-test/xwiki-platform-messagestream-test-tests
./xwiki-platform-core/xwiki-platform-vfs/xwiki-platform-vfs-test/xwiki-platform-vfs-test-tests
./xwiki-platform-core/xwiki-platform-office/xwiki-platform-office-test/xwiki-platform-office-test-tests
./xwiki-platform-core/xwiki-platform-wysiwyg/xwiki-platform-wysiwyg-test/xwiki-platform-wysiwyg-test-tests
vmassol ~/dev/xwiki/xwiki-platform (master *+) $

There are 31.

6

Sure but why is that a problem? There’s one test that is parametrized in the distribution tests, it’s xwiki-platform-distribution-flavor-test-escaping and it generates lots of tests.

7

Actually it’s the opposite ATM, we have plenty of WVAG-related failures that we need to fix.

Could you explain what yo mean by “case2) test cases fail a lot when compared to case 1).”? Do you have examples?

No you cannot. Both types of tests are need to cover the features of XWiki. If you take only cases 1) tests then you’ll be missing a large option of XWiki. Now it depends what your study is about.

If you want to see details of what is covered by each test you can check XWiki Platform - Parent POM 10.9-SNAPSHOT

Also note that case1) tests use some old selenium version, see https://dev.xwiki.org/xwiki/bin/view/Community/Testing#HOldSelenium1-basedFramework

8

Thanks for your reply.

I am researching how to decompose a software system into several service candidates. The concrete scenario is:
when some business functionalities of an application are determined to be extracted out as several services, we aim to design appraoch to recommand promising service candidates to assis the splitting or extraction procedure.

In my experiment, the functional test suites are required, since each functional test case presents a business use case. I believe that the execution traces under funtional tests can help to figure out which methods or classes are dedicated into a similar business functionality.

How do you think the idea or other aspects about my research?

Thanks.:slight_smile:

9

  1. I am sorry for my ambigous expression of “case2) test cases fail a lot when compared to case 1)”. My original meaning is that:
    the number of functional test cases in respective modules is smaller than that in flavor package. Even though the number is smaller in the former one, there are a few tests fail when i run them. I think it is ok and normal.

  2. I have used the functional tests based on your specification. Your reply really helps me a lot.
    I really appreciate it.