You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Jun 04, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3400996340,14cyclictest35966irq/58-eth021:45:113
3369996325,24cyclictest131rcu_preempt20:40:090
337999593,45cyclictest23568-21ssh22:20:131
337999575,43cyclictest29137-21ssh22:05:131
337999501,3cyclictest31017-21ssh00:30:131
337999491,43cyclictest3937-21ssh22:45:121
336999474,28cyclictest3918-21ssh00:15:150
336999471,34cyclictest2026-21ssh23:20:110
339199463,32cyclictest16425-21ssh21:40:112
339199453,31cyclictest18240-21ssh00:05:122
337999454,36cyclictest20087-21ssh22:00:131
340099441,22cyclictest35966irq/58-eth022:10:143
336999431,35cyclictest5633-21ssh23:40:130
336999431,35cyclictest5633-21ssh23:40:120
336999431,34cyclictest27312-21ssh22:40:120
336999431,34cyclictest27312-21ssh22:40:110
3400994227,11cyclictest35966irq/58-eth023:20:123
339199424,20cyclictest27-21ksoftirqd/223:55:122
337999422,35cyclictest2828-21kthreadcore21:10:191
337999421,33cyclictest18235-21ssh00:05:131
3379994211,22cyclictest21-21ksoftirqd/120:10:081
337999411,3cyclictest0-21swapper/123:05:111
340099406,22cyclictest35966irq/58-eth000:20:143
340099404,21cyclictest35966irq/58-eth022:00:133
339199401,31cyclictest0-21swapper/223:30:122
3400993923,9cyclictest35966irq/58-eth022:20:143
337999391,5cyclictest32643-21ssh22:25:131
3379993913,20cyclictest21-21ksoftirqd/100:25:131
336999396,15cyclictest131rcu_preempt21:31:390
336999392,3cyclictest131rcu_preempt22:25:120
336999391,29cyclictest9317-21ssh22:30:140
3400993822,10cyclictest35966irq/58-eth023:30:133
337999382,4cyclictest0-21swapper/121:50:141
337999380,8cyclictest24017-21ssh21:43:481
336999384,22cyclictest12-21ksoftirqd/020:00:090
337999379,24cyclictest3129-21cat21:30:291
337999372,6cyclictest0-21swapper/120:30:091
337999372,24cyclictest14612-21ssh23:45:141
337999371,6cyclictest21824-21apt-get22:55:101
337999370,8cyclictest8890-21latency_hist00:00:021
336999374,24cyclictest9123-21ssh00:00:100
339199363,25cyclictest0-21swapper/222:06:452
337999362,9cyclictest29211-21sed20:00:131
337999361,5cyclictest25410-21ssh21:45:101
3379993610,17cyclictest21-21ksoftirqd/119:10:121
340099359,19cyclictest35966irq/58-eth022:30:133
340099353,20cyclictest35966irq/58-eth021:40:113
339199353,24cyclictest2045-21ssh23:20:122
339199352,22cyclictest12934-21ssh22:50:142
3379993516,9cyclictest630-21Async09:36:311
337999351,4cyclictest0-21swapper/100:20:131
3379993513,9cyclictest630-21Async09:36:311
337999351,25cyclictest5643-21ssh23:40:141
337999351,25cyclictest5643-21ssh23:40:131
337999351,10cyclictest12527-21tr00:20:001
337999350,8cyclictest18573-21ssh22:51:311
336999358,7cyclictest131rcu_preempt21:40:120
336999351,26cyclictest12937-21ssh00:20:140
337999348,23cyclictest5245-21cron22:10:031
337999341,9cyclictest27616-21ssh22:40:161
337999341,9cyclictest27616-21ssh22:40:151
337999341,9cyclictest16619-21latency_hist20:40:021
337999341,8cyclictest28334-21kthreadcore00:10:191
337999341,6cyclictest18092-21latency_hist22:35:011
337999341,5cyclictest0-21swapper/119:20:081
337999340,8cyclictest12144-21ssh21:19:411
336999341,28cyclictest12801-21ssh21:20:100
336999341,27cyclictest20022-21ssh23:30:140
340099334,18cyclictest35966irq/58-eth023:35:133
339199331,24cyclictest7408-21ssh23:05:122
337999332,8cyclictest20278-21grep23:30:161
337999332,8cyclictest18594-21sed19:40:011
337999331,9cyclictest18329-21latency_hist19:35:021
337999331,8cyclictest15007-21timerwakeupswit23:25:261
337999331,8cyclictest13779-21kthreadcore21:20:161
3379993310,20cyclictest26913-21cron22:40:001
336999333,18cyclictest12-21ksoftirqd/020:50:090
336999331,4cyclictest131rcu_preempt19:20:080
340099328,14cyclictest35966irq/58-eth022:45:113
340099327,15cyclictest35966irq/58-eth023:25:113
340099325,13cyclictest35966irq/58-eth023:45:143
340099324,17cyclictest35966irq/58-eth021:55:123
339199322,23cyclictest14606-21ssh22:15:152
339199321,24cyclictest7477-21ssh21:35:142
337999321,9cyclictest12355-21kthreadcore21:55:191
337999321,8cyclictest28833-21latency_hist20:00:021
337999321,8cyclictest25300-21kthreadcore23:50:191
337999321,7cyclictest2018-21apt-get23:20:121
337999321,6cyclictest9348-21fschecks_time00:00:131
337999321,5cyclictest6698-21diskstats20:20:101
337999321,4cyclictest11246-21ntp_states00:35:261
336999323,17cyclictest12-21ksoftirqd/021:00:090
340099316,14cyclictest35966irq/58-eth022:55:113
339199315,18cyclictest27346irq/40-dwc2_hso19:10:132
337999313,3cyclictest29051-21kworker/u8:219:40:051
337999311,9cyclictest23522-21awk19:50:011
337999311,8cyclictest32554-21cat23:35:241
337999311,8cyclictest28422-21latency_hist19:55:021
337999311,8cyclictest28422-21latency_hist19:55:011
337999311,8cyclictest19989-21missed_timers20:40:181
337999311,7cyclictest28884-21kthreadcore23:15:221
337999311,10cyclictest30897-21ssh23:00:141
337999310,7cyclictest23635-21cut21:25:211
336999312,25cyclictest27275-21ssh00:10:120
3369993118,8cyclictest12-21ksoftirqd/023:55:110
340099309,12cyclictest35966irq/58-eth023:05:133
340099302,19cyclictest35966irq/58-eth020:30:093
3400993012,13cyclictest35966irq/58-eth000:25:133
339199303,21cyclictest0-21swapper/222:42:382
339199303,21cyclictest0-21swapper/222:42:372
3391993021,4cyclictest0-21swapper/200:30:142
339199301,17cyclictest11031-21ssh21:55:142
337999304,3cyclictest10095-1kworker/u9:021:10:021
337999301,8cyclictest26727-21acpi21:00:081
337999301,6cyclictest16419-21cat23:10:121
337999301,5cyclictest11410-21munin-plugin-st20:30:021
337999300,8cyclictest21645-21latency_hist20:50:021
337999300,7cyclictest22094-21sed20:50:131
336999302,22cyclictest2548-21ssh22:06:440
3400992911,13cyclictest35966irq/58-eth023:00:143
339199293,20cyclictest0-21swapper/221:26:422
339199292,20cyclictest0-21swapper/220:50:102
339199292,20cyclictest0-21swapper/220:20:092
339199291,21cyclictest16438-21rm23:10:122
337999291,8cyclictest3469-21latency_hist19:10:031
337999291,7cyclictest13386-21idleruntime-cro19:30:001
337999291,6cyclictest1462-21latency_hist20:10:021
337999290,9cyclictest26559-21latency_hist21:00:021
336999297,16cyclictest12-21ksoftirqd/000:35:130
336999291,22cyclictest23644-21ssh23:50:130
339199282,19cyclictest0-21swapper/222:55:132
337999281,5cyclictest8339-21cron19:20:011
340099273,15cyclictest0-21swapper/321:51:363
340099272,15cyclictest35966irq/58-eth000:00:103
339199275,12cyclictest27346irq/40-dwc2_hso22:10:132
339199274,16cyclictest27346irq/40-dwc2_hso21:45:122
339199272,16cyclictest27346irq/40-dwc2_hso00:00:112
336999273,17cyclictest0-21swapper/019:30:080
336999272,4cyclictest30991-21apt-get00:30:140
340099263,14cyclictest0-21swapper/322:06:443
340099261,13cyclictest35966irq/58-eth021:30:113
339199262,18cyclictest20712-21sshd00:23:582
339199262,10cyclictest7430-21missed_timers22:45:242
339199261,17cyclictest23635-21ssh23:50:132
336999262,18cyclictest7709-21sh21:51:360
336999261,20cyclictest21827-21ssh22:55:110
340099253,14cyclictest35966irq/58-eth022:50:143
3400992512,9cyclictest35966irq/58-eth000:30:133
339199253,6cyclictest30875-21ssh21:30:132
339199253,15cyclictest27346irq/40-dwc2_hso19:30:082
339199252,8cyclictest29516-21ssh22:21:462
339199252,16cyclictest27346irq/40-dwc2_hso20:40:092
339199251,9cyclictest25870-21ssh22:01:352
3391992513,5cyclictest0-21swapper/200:25:522
339199251,16cyclictest27346irq/40-dwc2_hso20:00:082
339199251,13cyclictest12804-21sh21:20:112
337999251,8cyclictest6383-21sendmail20:20:011
339199249,7cyclictest27-21ksoftirqd/222:34:462
339199247,9cyclictest131rcu_preempt00:11:402
336999243,3cyclictest131rcu_preempt23:00:140
339199237,10cyclictest131rcu_preempt21:16:132
339199236,3cyclictest27-21ksoftirqd/223:45:272
339199236,10cyclictest12641-21diskmemload00:20:022
339199232,16cyclictest0-21swapper/223:26:552
339199232,16cyclictest0-21swapper/222:29:382
339199232,16cyclictest0-21swapper/221:50:362
339199232,16cyclictest0-21swapper/221:13:372
3391992315,5cyclictest1766-21ssh23:20:022
3391992313,8cyclictest0-21swapper/222:35:232
3391992311,5cyclictest16116-21ssh00:39:562
3369992311,7cyclictest0-21swapper/021:25:130
340099224,10cyclictest35966irq/58-eth021:35:153
339199227,4cyclictest131rcu_preempt23:35:042
339199226,10cyclictest27-21ksoftirqd/223:02:042
339199224,2cyclictest131rcu_preempt20:30:262
339199223,16cyclictest23381-21latency_hist19:45:012
3391992212,7cyclictest27-21ksoftirqd/219:40:012
339199221,12cyclictest5412-21head20:10:242
3391992210,9cyclictest5581-21ssh23:40:112
3391992210,9cyclictest5581-21ssh23:40:102
336999223,13cyclictest0-21swapper/019:50:090
336999223,13cyclictest0-21swapper/019:50:080
336999222,14cyclictest17679-21ssh23:27:390
3369992216,3cyclictest12-21ksoftirqd/023:10:130
336999221,17cyclictest25452-21ssh23:15:140
340099214,10cyclictest0-21swapper/322:35:113
340099212,13cyclictest35966irq/58-eth023:15:123
340099212,13cyclictest35966irq/58-eth023:10:113
339199218,9cyclictest11407-21sh20:30:012
339199211,16cyclictest16455-21sendmail-msp20:40:012
339199211,10cyclictest27757-21kthreadcore21:00:162
336999211,7cyclictest32640-21ntp_192.168.11423:35:250
336999211,2cyclictest131rcu_preempt21:35:260
3369992112,6cyclictest0-21swapper/022:15:160
340099202,13cyclictest0-21swapper/300:15:133
339199203,14cyclictest23543-21apt-key19:50:012
339199201,9cyclictest25170-21kthreadcore19:50:152
339199201,9cyclictest25170-21kthreadcore19:50:152
339199201,8cyclictest12375-21unixbench_multi19:20:252
339199201,15cyclictest13524-21latency_hist19:30:012
339199201,10cyclictest8438-21latency_hist19:20:012
336999206,9cyclictest27537-21ssh22:03:360
336999202,13cyclictest0-21swapper/021:55:360
336999202,10cyclictest31534-21diskmemload21:10:100
3369992013,4cyclictest23835-21sh22:20:180
336999201,15cyclictest18295-21ssh22:35:110
340099199,6cyclictest0-21swapper/300:14:463
340099195,4cyclictest35966irq/58-eth022:25:113
340099193,11cyclictest0-21swapper/322:15:363
340099193,11cyclictest0-21swapper/321:26:433
3400991915,2cyclictest35966irq/58-eth023:40:133
3400991915,2cyclictest35966irq/58-eth023:40:123
3400991911,5cyclictest0-21swapper/300:35:283
3400991910,7cyclictest0-21swapper/321:15:373
339199195,10cyclictest0-21swapper/220:10:012
339199191,11cyclictest31532-21apt21:10:012
339199191,10cyclictest26478-21cron21:00:012
339199190,10cyclictest6373-21idleruntime-cro20:20:002
336999199,7cyclictest27726-21cut21:45:200
336999191,6cyclictest16071-21awk23:10:020
336999191,5cyclictest0-21swapper/000:25:060
336999191,15cyclictest3263-21kthreadcore20:10:170
336999191,14cyclictest18255-21ssh00:05:130
336999191,14cyclictest15316-21irqstats23:45:190
3369991911,4cyclictest1-21systemd22:46:500
3369991910,6cyclictest1-21systemd22:11:450
340099189,6cyclictest0-21swapper/321:10:413
340099185,10cyclictest0-21swapper/322:40:133
340099185,10cyclictest0-21swapper/322:40:123
340099182,10cyclictest0-21swapper/320:50:093
340099182,10cyclictest0-21swapper/320:10:103
340099181,13cyclictest0-21swapper/300:05:233
340099181,11cyclictest0-21swapper/323:50:153
3400991810,5cyclictest35966irq/58-eth021:20:103
339199182,8cyclictest3394-21latency_hist19:10:012
339199181,8cyclictest28632-21sendmail20:00:012
339199181,13cyclictest21460-21cron20:50:002
336999183,11cyclictest12924-21ssh22:50:130
3369991812,4cyclictest0-21swapper/020:30:250
340099179,6cyclictest0-21swapper/323:56:313
340099172,9cyclictest0-21swapper/319:10:133
340099171,13cyclictest0-21swapper/320:20:123
336999178,7cyclictest3613-21diskstats19:10:160
336999176,3cyclictest131rcu_preempt21:20:000
336999171,9cyclictest11177-21munin-run20:25:010
336999171,8cyclictest13374-21sh19:30:010
336999171,7cyclictest1337-21awk20:10:010
340099162,10cyclictest0-21swapper/319:40:043
340099161,9cyclictest0-21swapper/320:00:243
336999169,5cyclictest6356-21cron20:20:000
336999168,6cyclictest21439-21sshd20:45:360
336999163,9cyclictest16534-21latency_hist20:40:010
336999161,4cyclictest0-21swapper/020:59:390
336999161,3cyclictest0-21swapper/019:47:190
336999161,3cyclictest0-21swapper/019:43:450
340099158,3cyclictest0-21swapper/319:20:243
340099151,9cyclictest0-21swapper/319:50:103
340099151,9cyclictest0-21swapper/319:50:093
340099151,12cyclictest0-21swapper/319:30:233
340099151,10cyclictest0-21swapper/321:00:153
336999151,11cyclictest31578-21latency_hist21:10:010
336999151,11cyclictest28616-21munin-plugin-st20:00:000
336999151,11cyclictest18383-21cron19:40:020
336999151,10cyclictest8375-21latency_hist19:20:010
340099149,3cyclictest0-21swapper/320:18:283
336999141,10cyclictest11536-21latency_hist20:30:020
336999135,6cyclictest3399-21date19:10:010
340099127,3cyclictest0-21swapper/320:39:573
340099123,7cyclictest0-21swapper/321:10:023
340099122,7cyclictest0-21swapper/319:50:023
340099121,9cyclictest0-21swapper/320:40:223
340099121,8cyclictest0-21swapper/321:00:013
340099121,7cyclictest0-21swapper/320:00:003
340099121,4cyclictest0-21swapper/320:50:013
340099116,3cyclictest0-21swapper/319:18:263
340099112,5cyclictest0-21swapper/320:05:053
340099111,7cyclictest0-21swapper/319:30:023
340099101,7cyclictest0-21swapper/319:37:373
340099101,6cyclictest0-21swapper/320:30:013
34009991,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional