You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 21:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 08, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1357699521,3cyclictest131rcu_preempt09:50:110
13609995114,26cyclictest35966irq/58-eth010:00:123
1359699502,33cyclictest4660-21ssh10:25:122
1360999494,28cyclictest35966irq/58-eth012:00:143
13585994819,3cyclictest131rcu_preempt08:15:121
13585994819,3cyclictest131rcu_preempt08:15:121
1357699452,9cyclictest131rcu_preempt08:50:140
1359699441,36cyclictest0-21swapper/210:05:132
1359699431,24cyclictest13569-21ssh10:30:122
13596994235,4cyclictest2594-21ssh11:00:152
1358599421,33cyclictest835-21ssh12:30:131
13596993825,10cyclictest13584-21ssh12:18:162
1360999371,20cyclictest35966irq/58-eth012:15:133
1359699374,26cyclictest31876-21diskmemload11:20:152
1359699371,25cyclictest6793-21ssh09:50:112
1359699351,29cyclictest930-21snmpd11:45:142
1357699351,27cyclictest32630-21ssh11:35:150
1357699351,27cyclictest32630-21ssh11:35:150
1357699341,22cyclictest8165-21ssh10:45:130
1359699339,20cyclictest27-21ksoftirqd/208:35:142
1359699334,21cyclictest27346irq/40-dwc2_hso09:00:122
1359699332,22cyclictest28171-21ssh10:20:112
1359699331,27cyclictest24352-21ssh12:25:142
1357699331,24cyclictest0-21swapper/007:30:130
1360999321,17cyclictest35966irq/58-eth012:20:143
1359699323,12cyclictest24694-21ssh10:00:132
1360999303,15cyclictest0-21swapper/311:50:123
1359699308,11cyclictest27-21ksoftirqd/211:55:132
1359699303,20cyclictest0-21swapper/209:25:132
13596993020,5cyclictest0-21swapper/209:10:132
1357699301,21cyclictest29287-21ssh11:15:120
1360999291,18cyclictest35966irq/58-eth009:35:133
1360999291,13cyclictest35966irq/58-eth009:45:133
1357699291,19cyclictest21126-21rm12:05:150
13576992911,12cyclictest12-21ksoftirqd/012:20:160
1360999288,15cyclictest35966irq/58-eth012:30:133
1360999281,15cyclictest35966irq/58-eth008:05:133
1359699282,24cyclictest23343-21ssh12:24:162
1359699282,17cyclictest27346irq/40-dwc2_hso07:50:122
13585992813,4cyclictest0-21swapper/109:55:261
1359699272,15cyclictest930-21snmpd11:10:122
1359699272,15cyclictest27346irq/40-dwc2_hso08:00:112
1359699272,14cyclictest12294-21ssh09:35:142
13596992712,9cyclictest261rcuc/210:50:132
1359699271,20cyclictest135-21systemd-journal09:20:012
1359699271,13cyclictest27-21ksoftirqd/211:32:012
1359699271,13cyclictest27-21ksoftirqd/211:32:012
13585992717,4cyclictest131rcu_preempt07:45:111
1357699272,19cyclictest8765-21ssh12:34:160
13609992610,9cyclictest35966irq/58-eth011:45:153
1359699266,14cyclictest1-21systemd07:40:132
1359699265,6cyclictest131rcu_preempt07:45:272
1359699263,19cyclictest11004-21kthreadcore12:35:192
1359699263,18cyclictest930-21snmpd08:05:152
1359699262,21cyclictest0-21swapper/210:45:132
1359699262,17cyclictest930-21snmpd07:35:142
1359699262,17cyclictest29288-21ssh11:15:122
1358599262,18cyclictest29745-21ssh12:26:121
13585992615,3cyclictest0-21swapper/109:51:131
1358599261,21cyclictest21274-21rm09:40:121
13576992612,11cyclictest31347-21cut10:40:020
13576992611,3cyclictest13293-21kthreadcore11:05:210
1359699256,7cyclictest27-21ksoftirqd/209:56:312
1359699256,12cyclictest27-21ksoftirqd/209:40:132
1359699252,15cyclictest27346irq/40-dwc2_hso08:10:112
1359699252,15cyclictest27346irq/40-dwc2_hso08:10:112
1359699251,20cyclictest930-21snmpd07:20:132
1359699251,12cyclictest31741-21kthreadcore09:45:192
1359699251,11cyclictest2603166ssh10:55:132
13596992510,11cyclictest1047-21fschecks_time12:30:162
1359699250,14cyclictest9317-21cut11:40:162
1358599252,17cyclictest12270-21ssh12:00:141
1357699251,16cyclictest26000-21sh10:55:120
1359699244,12cyclictest27346irq/40-dwc2_hso09:05:132
1359699243,14cyclictest27346irq/40-dwc2_hso08:25:132
1359699243,14cyclictest27346irq/40-dwc2_hso08:25:132
1359699242,17cyclictest930-21snmpd12:05:252
13596992413,4cyclictest0-21swapper/208:30:112
13596992412,9cyclictest0-21swapper/209:30:152
13596992412,9cyclictest0-21swapper/209:30:152
1359699241,19cyclictest6257-21ls12:15:022
1359699241,18cyclictest30463-21netstat11:50:232
1359699241,11cyclictest27-21ksoftirqd/210:38:432
13596992410,5cyclictest16569-21kthreadcore11:25:192
13596992410,5cyclictest16569-21kthreadcore11:25:192
13585992412,10cyclictest0-21swapper/108:35:221
13585992411,10cyclictest21-21ksoftirqd/111:15:121
1357699245,3cyclictest131rcu_preempt10:00:170
1357699242,17cyclictest151rcuc/012:25:140
1357699241,15cyclictest32016-21awk10:20:270
1357699241,14cyclictest23643-21ssh11:30:120
1357699241,14cyclictest23643-21ssh11:30:120
1357699241,13cyclictest0-21swapper/010:15:240
1360999238,9cyclictest35966irq/58-eth009:40:123
1359699238,3cyclictest261rcuc/211:35:132
1359699238,3cyclictest261rcuc/211:35:132
1359699234,5cyclictest27-21ksoftirqd/212:00:322
13596992311,4cyclictest0-21swapper/210:13:032
13596992311,4cyclictest0-21swapper/210:13:032
13596992310,4cyclictest27-21ksoftirqd/210:15:282
1358599232,9cyclictest131rcu_preempt11:35:171
1358599232,9cyclictest131rcu_preempt11:35:171
1358599232,4cyclictest131rcu_preempt10:20:381
1358599232,3cyclictest131rcu_preempt09:25:171
1358599232,3cyclictest131rcu_preempt08:25:131
1358599232,3cyclictest131rcu_preempt08:25:131
13585992314,6cyclictest21-21ksoftirqd/110:00:111
1358599231,18cyclictest9782-21ssh12:35:141
1357699231,6cyclictest3137-21ssh10:05:210
1357699231,5cyclictest131rcu_preempt09:45:120
1357699231,13cyclictest21265-21diskmemload09:40:110
13576992310,5cyclictest12-21ksoftirqd/007:35:160
1357699230,17cyclictest18028-21ssh11:45:140
1360999223,12cyclictest0-21swapper/312:26:123
1359699228,11cyclictest12940-21df_inode08:50:162
1359699224,13cyclictest27346irq/40-dwc2_hso08:40:132
1359699223,8cyclictest167-21systemd-network07:15:132
1359699223,3cyclictest27-21ksoftirqd/209:24:202
1359699223,12cyclictest27346irq/40-dwc2_hso07:55:132
1359699221,9cyclictest1436-21kthreadcore10:40:222
1359699221,12cyclictest15967-21cron08:20:012
1359699221,12cyclictest15967-21cron08:20:012
1359699221,11cyclictest11508-21sh11:05:132
13596992210,10cyclictest541-21sshd07:30:132
1358599226,3cyclictest131rcu_preempt12:15:231
13585992217,3cyclictest2608-21ssh11:00:161
13585992214,5cyclictest0-21swapper/107:10:281
13585992212,7cyclictest22566-21diskmemload10:35:151
13585992212,7cyclictest11501-21ssh11:05:131
13585992211,8cyclictest0-21swapper/111:50:131
1358599221,17cyclictest3342-21ssh11:55:131
1358599221,15cyclictest17099-21ssh10:50:131
1357699227,8cyclictest131rcu_preempt07:25:010
1357699225,7cyclictest131rcu_preempt08:25:160
1357699225,7cyclictest131rcu_preempt08:25:160
1357699223,7cyclictest5657-21ssh10:25:190
1357699222,15cyclictest10182-21apt-get07:40:130
1357699222,13cyclictest31876-21diskmemload12:15:160
1357699221,6cyclictest23305-21ssh11:10:210
1357699221,5cyclictest0-21swapper/010:31:530
1357699221,5cyclictest0-21swapper/009:25:190
1357699221,4cyclictest0-21swapper/009:21:400
13576992214,5cyclictest9642-21mailstats11:20:290
1357699221,17cyclictest13917-21grep07:10:170
1357699221,16cyclictest30291-21irqcore12:10:170
1360999211,11cyclictest35966irq/58-eth009:50:123
13609992110,9cyclictest0-21swapper/310:06:553
1359699212,12cyclictest0-21swapper/208:55:122
13596992112,7cyclictest0-21swapper/208:45:162
1359699211,11cyclictest18267-21cron07:15:012
1358599219,3cyclictest21-21ksoftirqd/109:15:121
1358599214,12cyclictest131rcu_preempt11:40:141
1358599212,2cyclictest131rcu_preempt11:27:211
1358599212,2cyclictest131rcu_preempt11:27:211
1358599212,2cyclictest131rcu_preempt09:21:411
1358599211,9cyclictest0-21swapper/111:45:221
1358599211,8cyclictest6082-21basename12:15:011
13585992113,5cyclictest8123-21/usr/sbin/munin10:45:111
13585992112,5cyclictest19777-21usage_Anybus12:20:311
13585992111,7cyclictest7101-21run-parts09:30:261
13585992111,7cyclictest7101-21run-parts09:30:261
1358599211,17cyclictest1262-21ssh10:05:141
13585992110,7cyclictest20418-21apt-get11:10:111
1358599210,3cyclictest131rcu_preempt10:41:271
1357699216,9cyclictest131rcu_preempt11:50:220
1357699214,4cyclictest17809-21ssh10:50:190
1357699212,3cyclictest0-21swapper/007:25:230
1357699212,16cyclictest11385-21apt-get08:15:120
1357699212,16cyclictest11385-21apt-get08:15:120
1357699211,9cyclictest11436-21kthreadcore09:15:210
1357699211,6cyclictest0-21swapper/009:13:400
1357699211,5cyclictest0-21swapper/007:15:220
1357699211,15cyclictest8340-21tr08:45:170
1357699211,14cyclictest31876-21diskmemload11:40:140
1357699211,11cyclictest19459-21ntp_states09:55:270
1357699211,10cyclictest12219-21/usr/sbin/munin12:00:250
1357699210,8cyclictest26321-21grep09:00:280
1357699210,10cyclictest1902-21fschecks_count08:05:170
1357699210,10cyclictest18487-21ps07:45:260
1360999203,11cyclictest0-21swapper/312:13:163
1360999202,13cyclictest35966irq/58-eth009:20:123
13609992012,6cyclictest0-21swapper/311:33:113
13609992012,6cyclictest0-21swapper/311:33:113
1360999201,10cyclictest35966irq/58-eth012:05:143
1360999200,12cyclictest35966irq/58-eth010:50:133
1359699201,10cyclictest16433-21fschecks_count08:20:152
1358599208,9cyclictest0-21swapper/108:30:141
1358599202,3cyclictest131rcu_preempt08:50:151
1358599202,2cyclictest131rcu_preempt07:50:151
1358599202,13cyclictest24620-21memory12:05:241
1358599201,8cyclictest23833-21kthreadcore09:00:191
13585992013,4cyclictest30639-21if_eth109:45:161
1358599201,2cyclictest131rcu_preempt11:22:131
1358599201,2cyclictest131rcu_preempt10:25:251
1358599201,10cyclictest32242-21/usr/sbin/munin09:10:151
13585992010,7cyclictest0-21swapper/107:40:241
1358599200,2cyclictest131rcu_preempt10:33:111
1358599200,13cyclictest1767-21find08:05:131
1357699207,7cyclictest4593-21grep11:00:230
1357699201,9cyclictest5561-21kthreadcore08:40:210
1357699201,9cyclictest31876-21diskmemload09:30:120
1357699201,9cyclictest31876-21diskmemload09:30:120
1357699201,9cyclictest0-21swapper/009:35:290
1357699201,6cyclictest32458-21kthreadcore08:00:210
1357699201,5cyclictest12543-21ssh10:10:210
1357699201,5cyclictest12543-21ssh10:10:210
1357699201,3cyclictest0-21swapper/011:56:020
1357699201,14cyclictest9781-21sh12:35:140
1357699201,10cyclictest1731-21kthreadcore10:40:220
13576992010,7cyclictest27326-21sshd09:05:130
1357699200,17cyclictest21287-21switchtime08:55:270
1360999192,10cyclictest0-21swapper/308:20:043
1360999192,10cyclictest0-21swapper/307:20:133
1360999191,12cyclictest0-21swapper/311:40:163
13609991910,7cyclictest0-21swapper/310:28:453
13609991910,6cyclictest35966irq/58-eth011:00:143
13609991910,6cyclictest35966irq/58-eth010:45:133
13609991910,6cyclictest0-21swapper/310:24:333
13609991910,6cyclictest0-21swapper/309:26:063
1359699191,9cyclictest13639-21latency_hist07:10:012
1359699191,10cyclictest28814-21irqrtprio07:25:192
1358599191,9cyclictest11855-21kthreadcore10:10:201
1358599191,9cyclictest11855-21kthreadcore10:10:201
13585991913,4cyclictest19143-21/usr/sbin/munin10:15:151
1358599191,16cyclictest25323-21kthreadcore07:20:211
13585991910,6cyclictest0-21swapper/109:38:001
1358599190,16cyclictest24557-21find07:55:131
1358599190,16cyclictest22291-21ls09:00:011
1357699191,7cyclictest21982-21kthreadcore07:50:200
1357699191,5cyclictest6805-21grep08:10:150
1357699191,5cyclictest6805-21grep08:10:150
1357699190,6cyclictest16005-21tr11:25:180
1357699190,6cyclictest16005-21tr11:25:180
1360999183,10cyclictest0-21swapper/312:36:133
13609991812,4cyclictest0-21swapper/311:06:313
13609991811,4cyclictest0-21swapper/311:20:153
1360999181,12cyclictest0-21swapper/310:10:163
1360999181,12cyclictest0-21swapper/310:10:163
1360999181,11cyclictest0-21swapper/307:15:133
1360999181,10cyclictest0-21swapper/308:55:143
1360999181,10cyclictest0-21swapper/308:35:143
13609991810,5cyclictest35966irq/58-eth010:15:123
1358599189,7cyclictest21-21ksoftirqd/108:00:111
1358599181,3cyclictest131rcu_preempt07:35:161
13585991812,4cyclictest0-21swapper/109:05:191
1358599181,14cyclictest11333-21latency_hist08:15:011
1358599181,14cyclictest11333-21latency_hist08:15:011
1358599180,8cyclictest1391-21kthreadcore07:30:191
1357699182,2cyclictest131rcu_preempt07:55:170
1357699181,7cyclictest28166-21kthreadcore08:30:190
1357699181,4cyclictest131rcu_preempt07:10:010
13576991811,4cyclictest854-21sshd08:35:130
1357699180,9cyclictest16986-21kthreadcore08:20:190
1360999179,6cyclictest0-21swapper/310:57:443
1360999179,6cyclictest0-21swapper/309:31:273
1360999179,6cyclictest0-21swapper/309:31:273
1360999179,6cyclictest0-21swapper/309:11:503
1360999179,3cyclictest0-21swapper/311:11:423
1360999178,6cyclictest0-21swapper/310:30:223
1360999178,5cyclictest35966irq/58-eth011:35:133
1360999178,5cyclictest35966irq/58-eth011:35:133
1360999172,10cyclictest0-21swapper/309:05:143
1360999172,10cyclictest0-21swapper/307:25:153
1360999171,12cyclictest0-21swapper/307:10:173
1360999171,10cyclictest0-21swapper/308:10:133
1360999171,10cyclictest0-21swapper/308:10:133
13609991710,4cyclictest0-21swapper/309:55:303
1358599173,11cyclictest9677-21kthreadcore08:45:191
1358599171,13cyclictest29919-21ls10:55:291
1358599171,12cyclictest3096-21apt-get08:40:131
1358599171,12cyclictest23609-21/usr/sbin/munin11:30:151
1358599171,12cyclictest23609-21/usr/sbin/munin11:30:151
1358599171,12cyclictest13614-21latency_hist07:10:011
1360999168,6cyclictest0-21swapper/311:55:443
1360999168,3cyclictest0-21swapper/310:42:363
1360999168,3cyclictest0-21swapper/310:35:453
1360999162,9cyclictest0-21swapper/307:30:133
1360999162,10cyclictest0-21swapper/311:15:123
1360999161,7cyclictest0-21swapper/309:20:023
13609991611,3cyclictest0-21swapper/311:27:233
13609991611,3cyclictest0-21swapper/311:27:233
1360999161,12cyclictest0-21swapper/308:30:013
1360999161,12cyclictest0-21swapper/308:30:013
1360999161,10cyclictest0-21swapper/308:40:143
1358599167,6cyclictest614-21dbus-daemon07:15:101
1358599163,9cyclictest0-21swapper/107:25:151
1358599161,5cyclictest0-21swapper/108:23:251
1360999151,10cyclictest0-21swapper/307:50:133
1360999148,4cyclictest33-21ksoftirqd/308:45:173
1360999141,7cyclictest0-21swapper/308:00:223
1360999141,7cyclictest0-21swapper/307:50:013
1360999141,10cyclictest0-21swapper/308:50:233
1360999131,10cyclictest0-21swapper/308:15:283
1360999131,10cyclictest0-21swapper/308:15:283
1360999131,10cyclictest0-21swapper/307:40:213
1360999128,2cyclictest33-21ksoftirqd/307:35:273
1360999127,3cyclictest0-21swapper/307:55:163
1360999121,8cyclictest0-21swapper/308:30:183
1360999111,6cyclictest0-21swapper/309:00:173
1360999101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional