ONOS
  1. ONOS
  2. ONOS-5691

Intents stuck in INSTALLING state with FlowObjective intent compiler when using "push-test-intents" with large batch size

    Details

    • Type: Bug Bug
    • Status: Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.0, 1.9.0
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Environment:

      commit 382243012ac88b3682e1267d34d8567e48596bc2

    • Story Points:
      3
    • Epic Link:
    • Sprint:
      Junco Sprint #1 - Platform, Junco Sprint #2 - Platform

      Description

      We noticed some failures in intent install/withdraw latency test when FlowObjective intent compiler is activated. Some intents stuck in INSTALLING state when using "push-test-intents" with a large batch size. This problem can be reproduced after gerrit change 10673.

      Steps for replication:
      1) Start single-node onos with "drivers, null"
      2) Configurations:

      cfg set org.onosproject.provider.nil.NullProviders deviceCount 7
      cfg set org.onosproject.provider.nil.NullProviders topoShape linear
      cfg set org.onosproject.provider.nil.NullProviders enabled true
      cfg set org.onosproject.net.intent.impl.IntentManager skipReleaseResourcesOnWithdrawal true
      cfg set org.onosproject.net.intent.impl.compiler.IntentConfigurableRegistrator useFlowObjectives true
      

      3) Repeat the following CLI commands

      onos> push-test-intents -i null:0000000000000001/6 null:0000000000000007/5 1000 1
      Failure: 2 intents not installed
      onos> push-test-intents -w null:0000000000000001/6 null:0000000000000007/5 1000 1
      Time to withdraw 1000 intents: 1458 ms
      onos> purge-intents
      

        Issue Links

          Activity

          Hide
          Jayasree Ghosh added a comment -

          I have tried push-test-intents with 1000, 1500 and 2000 intents.
          Intents installed successfully with 1000 and 1500. But with 2000 number of intents, it was failing for some random number of intents.
          push-test-intents -w null:0000000000000002/2 null:0000000000000003/2 2000 1
          Failure: 1999 intents not installed
          push-test-intents -i null:0000000000000001/6 null:0000000000000007/5 2000 2
          Failure: 1489 intents not installed

          Show
          Jayasree Ghosh added a comment - I have tried push-test-intents with 1000, 1500 and 2000 intents. Intents installed successfully with 1000 and 1500. But with 2000 number of intents, it was failing for some random number of intents. push-test-intents -w null:0000000000000002/2 null:0000000000000003/2 2000 1 Failure: 1999 intents not installed push-test-intents -i null:0000000000000001/6 null:0000000000000007/5 2000 2 Failure: 1489 intents not installed
          Hide
          Jon Hall added a comment -

          We think this was caused by the same change

          Show
          Jon Hall added a comment - We think this was caused by the same change
          Hide
          You Wang added a comment - - edited

          Besides applying gerrit change 10673, another way to trigger this symptom is to disable intent cleanup:

          cfg set org.onosproject.net.intent.impl.IntentCleanup enabled false
          

          But disabling intent cleanup doesn't help to reproduce the linked issue (ONOS-5683)

          Show
          You Wang added a comment - - edited Besides applying gerrit change 10673, another way to trigger this symptom is to disable intent cleanup: cfg set org.onosproject.net.intent.impl.IntentCleanup enabled false But disabling intent cleanup doesn't help to reproduce the linked issue ( ONOS-5683 )
          Hide
          You Wang added a comment -

          When the intents are stuck, all flows are actually installed.

          Show
          You Wang added a comment - When the intents are stuck, all flows are actually installed.
          Hide
          You Wang added a comment -

          Cannot be reproduced after applying both gerrit change 12165 and 12125.

          Show
          You Wang added a comment - Cannot be reproduced after applying both gerrit change 12165 and 12125.
          Hide
          Brian O'Connor added a comment -

          Updated the patch in Gerrit. Pier Luigi Ventre is going to rebase 12165 and after that, it would be great if you could retest.

          Show
          Brian O'Connor added a comment - Updated the patch in Gerrit. Pier Luigi Ventre is going to rebase 12165 and after that, it would be great if you could retest.
          Hide
          Pier Luigi Ventre added a comment -

          You Wang, done

          Show
          Pier Luigi Ventre added a comment - You Wang , done
          Hide
          You Wang added a comment -

          Now 12125 fixes this.

          Show
          You Wang added a comment - Now 12125 fixes this.

            People

            • Assignee:
              Pier Luigi Ventre
              Reporter:
              You Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile