You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-15 - 11:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Aug 15, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26015221697,95sleep00-21swapper/019:07:480
1838521540,1sleep30-21swapper/322:48:493
259762143107,25sleep30-21swapper/319:07:193
258582140105,24sleep20-21swapper/219:05:452
261282139103,25sleep10-21swapper/119:09:171
3233221140,4sleep30-21swapper/322:10:273
84942990,1sleep00-21swapper/022:17:150
97312690,2sleep10-21swapper/100:17:221
276112610,2sleep327610-21ssh22:55:153
159892610,2sleep20-21swapper/223:58:172
259022590,9sleep228-21ksoftirqd/222:54:332
208612580,4sleep1211rcuc/121:15:161
290902560,4sleep129087-21df_inode21:45:131
95202550,1sleep10-21swapper/122:41:431
32362540,3sleep30-21swapper/320:45:163
238832530,1sleep30-21swapper/323:15:363
143272530,2sleep00-21swapper/023:09:480
210332520,2sleep00-21swapper/023:38:010
316952510,4sleep10-21swapper/119:20:201
271542480,2sleep10-21swapper/121:43:531
218072480,4sleep1211rcuc/122:26:591
252652470,1sleep00-21swapper/022:30:120
285982440,1sleep30-21swapper/321:20:233
77192430,1sleep20-21swapper/222:16:292
7682370,21sleep213096-21diskmemload21:24:412
219862370,3chrt22-21ksoftirqd/100:02:301
2646099354,7cyclictest19212-21sed00:25:013
2646099354,7cyclictest19212-21sed00:25:003
107582350,1sleep20-21swapper/221:31:062
47372330,2chrt0-21swapper/121:26:521
75522320,2sleep30-21swapper/300:15:243
2646099323,8cyclictest2791-21mailstats21:25:213
2646099290,5cyclictest0-21swapper/319:45:003
2646099290,5cyclictest0-21swapper/319:45:003
3302280,2sleep10-21swapper/120:40:001
2646099285,2cyclictest141rcu_preempt00:30:013
2646099283,8cyclictest32563-21sed00:10:173
2646099283,10cyclictest1992-21rm23:47:473
26460992816,7cyclictest52950irq/141-enp3s0-23:50:123
2646099280,9cyclictest1041-21/usr/sbin/munin00:35:133
2646099280,10cyclictest9314-21ssh21:54:423
2646099275,6cyclictest34-21ksoftirqd/323:26:383
2646099273,9cyclictest17063-21ssh21:12:123
2646099273,8cyclictest30209-21sed21:45:233
2646099271,10cyclictest26355-21ssh21:19:393
72982260,3chrt0-21swapper/119:40:141
72982260,3chrt0-21swapper/119:40:131
2646099266,6cyclictest34-21ksoftirqd/323:35:433
2646099266,6cyclictest34-21ksoftirqd/321:31:023
2646099263,7cyclictest26399-21latency_hist22:55:003
2646099263,6cyclictest537-21rm22:35:203
2646099262,9cyclictest27733-21chrt19:10:223
2646099262,9cyclictest18773-21ssh22:00:383
2646099262,7cyclictest34-21ksoftirqd/323:56:203
2646099260,9cyclictest23502-21ssh22:28:403
2646099260,6cyclictest34-21ksoftirqd/300:07:503
26458992614,7cyclictest52850irq/140-enp3s0-21:50:241
2645899261,24cyclictest0-21swapper/119:12:071
2645799261,24cyclictest0-21swapper/019:12:360
169372260,2sleep10-21swapper/121:35:301
2646099255,7cyclictest34-21ksoftirqd/322:17:573
2646099255,7cyclictest34-21ksoftirqd/321:42:213
2646099253,8cyclictest34-21ksoftirqd/323:04:563
2646099253,7cyclictest34-21ksoftirqd/323:06:333
2646099253,7cyclictest34-21ksoftirqd/322:06:203
2646099253,7cyclictest34-21ksoftirqd/322:06:193
2646099253,7cyclictest34-21ksoftirqd/320:30:113
2646099253,6cyclictest34-21ksoftirqd/322:22:383
2646099252,6cyclictest34-21ksoftirqd/321:55:193
26460992516,7cyclictest18025-21cpuspeed_turbos20:05:133
2646099250,11cyclictest10602-21sshd21:00:413
2646099250,10cyclictest28566-21rm23:43:453
2645999259,5cyclictest28-21ksoftirqd/223:52:512
170282250,2sleep10-21swapper/123:10:461
162212251,21sleep216237-21ssh00:22:072
162212251,21sleep216237-21ssh00:22:062
2646099246,6cyclictest34-21ksoftirqd/321:35:383
2646099243,8cyclictest34-21ksoftirqd/300:01:093
2646099243,6cyclictest34-21ksoftirqd/323:20:073
2646099242,6cyclictest30728-21/usr/sbin/munin20:35:243
26460992420,2cyclictest331rcuc/319:20:573
2646099241,6cyclictest13096-21diskmemload22:40:193
26460992410,12cyclictest15598-21grep23:10:153
2645999249,2cyclictest141rcu_preempt22:00:132
2645999246,15cyclictest28-21ksoftirqd/221:40:112
2645999242,16cyclictest640-21snmpd23:35:132
2645999240,23cyclictest0-21swapper/219:12:102
2645899243,14cyclictest640-21snmpd20:05:131
26457992413,8cyclictest30941-21df_inode22:10:140
2646099233,7cyclictest34-21ksoftirqd/320:41:263
2646099233,6cyclictest27755-21ssh00:30:233
2646099232,8cyclictest34-21ksoftirqd/320:15:203
26460992319,3cyclictest331rcuc/319:17:333
26460992318,3cyclictest331rcuc/320:10:023
26460992317,3cyclictest8790-21turbostat.cron21:00:003
2646099231,6cyclictest10894-21ssh23:30:203
2646099230,6cyclictest27442-21ssh22:30:523
2645999238,13cyclictest28-21ksoftirqd/222:58:032
26459992319,2cyclictest28-21ksoftirqd/223:30:002
26459992318,3cyclictest28-21ksoftirqd/221:40:002
26459992317,4cyclictest28-21ksoftirqd/223:24:592
2645899234,13cyclictest16648-21kworker/u8:000:10:111
26457992314,7cyclictest29629-21memory23:20:180
26457992312,9cyclictest1169-21fschecks_time00:35:160
26457992311,8cyclictest3269-21apt-get21:50:120
2646099223,7cyclictest34-21ksoftirqd/321:08:333
2646099223,5cyclictest848-21df19:25:143
2646099223,5cyclictest5108-21diskstats20:50:163
26460992218,3cyclictest331rcuc/320:04:543
26460992218,3cyclictest331rcuc/319:38:283
2645999229,7cyclictest28-21ksoftirqd/200:02:022
2645999227,4cyclictest28-21ksoftirqd/200:05:452
2645999226,3cyclictest141rcu_preempt21:45:122
2645999222,10cyclictest28-21ksoftirqd/223:10:262
26459992214,5cyclictest52750irq/139-enp3s0-23:05:122
26459992212,8cyclictest16509-21memory20:00:202
2645899229,9cyclictest13096-21diskmemload22:00:141
26458992216,4cyclictest10941-21cron21:04:591
216172220,2sleep00-21swapper/021:15:220
2646099211,8cyclictest27500-21processes20:25:223
26460992117,3cyclictest331rcuc/319:59:433
2645999218,5cyclictest28-21ksoftirqd/200:37:102
2645999217,12cyclictest28-21ksoftirqd/222:49:162
2645999215,3cyclictest141rcu_preempt23:03:012
26459992115,4cyclictest10567-21uptime21:00:252
26459992113,6cyclictest27183-21memory00:30:172
26459992112,7cyclictest31067-21tune2fs22:10:152
26459992112,7cyclictest13917-21df_inode19:55:142
26459992112,6cyclictest20941-21gltestperf21:15:162
26459992111,2cyclictest141rcu_preempt21:29:522
2645899212,15cyclictest75-21kswapd000:05:121
2645899211,8cyclictest5225-21gltestperf23:50:151
26458992113,6cyclictest560-21dbus-daemon20:55:011
26458992113,6cyclictest23085-21ntp_states20:15:221
2645799219,8cyclictest19593-21apt-get00:25:130
26457992113,6cyclictest29266-21if_err_enp3s019:15:190
26457992113,6cyclictest27997-21ntpq22:55:170
26457992113,6cyclictest2743-21cut20:45:130
26457992113,6cyclictest17640-21pluginstate20:05:010
26457992111,6cyclictest16957-21sort22:00:140
26457992110,7cyclictest31661-21apt-get00:10:090
2646099202,7cyclictest34-21ksoftirqd/319:31:053
26460992017,2cyclictest331rcuc/319:49:593
2646099201,6cyclictest34-21ksoftirqd/319:50:223
2645999209,8cyclictest30907-21cut20:35:152
2645999209,3cyclictest28-21ksoftirqd/200:14:002
2645999209,2cyclictest28-21ksoftirqd/222:38:582
2645999209,2cyclictest28-21ksoftirqd/200:27:162
2645999208,10cyclictest271rcuc/219:30:012
2645999207,4cyclictest28-21ksoftirqd/221:54:592
2645999207,11cyclictest4746-21/usr/sbin/munin20:50:172
2645999205,7cyclictest141rcu_preempt22:42:382
2645999203,4cyclictest0-21swapper/222:34:002
26459992014,4cyclictest25841-21sh23:41:022
26459992013,4cyclictest3316-21latency20:45:182
26459992013,4cyclictest15033-21sh23:33:482
26459992011,6cyclictest5366-21smartctl19:35:152
2645999201,11cyclictest0-21swapper/200:19:572
26459992010,7cyclictest22190-21cat23:15:112
26459992010,6cyclictest52750irq/139-enp3s0-22:25:132
2645999200,4cyclictest0-21swapper/221:59:332
2645899201,9cyclictest9440-21/usr/sbin/munin23:05:221
26458992014,5cyclictest22-21ksoftirqd/100:00:011
26458992014,4cyclictest5871-21processes19:35:211
26458992013,5cyclictest28308-21mailstats22:55:211
26458992012,6cyclictest4607-21cut19:35:001
26458992012,6cyclictest29866-21pluginstate23:45:001
26458992012,6cyclictest26358-21open_files22:30:211
26458992012,6cyclictest11116-21ntp_states21:55:211
26458992011,7cyclictest31357-21ntp_states23:45:221
2645899201,17cyclictest17576-21sh23:35:191
26458992010,7cyclictest16648-21kworker/u8:023:20:191
2645799202,11cyclictest11723-21ssh22:20:100
2645799201,9cyclictest3541-21sort23:25:140
26457992014,4cyclictest8204-21threads19:40:240
26457992014,4cyclictest8204-21threads19:40:230
26457992014,4cyclictest3252-21idleruntime19:30:140
26457992014,3cyclictest21316-21sh23:14:590
26457992013,4cyclictest8216-21timerwakeupswit20:55:240
26457992012,6cyclictest4239-21basename23:50:000
26457992012,6cyclictest23320-21python320:15:230
26457992012,6cyclictest13096-21diskmemload22:40:210
26457992011,7cyclictest7630-21seq21:29:430
2646099190,7cyclictest34-21ksoftirqd/320:23:433
2645999191,9cyclictest0-21swapper/222:05:122
2645999191,9cyclictest0-21swapper/222:05:112
26459991913,4cyclictest14745-21sh22:21:412
26459991912,5cyclictest2533-21date20:45:002
26459991911,6cyclictest31039-21expr19:20:132
26459991911,6cyclictest13878-21expr21:10:132
26459991911,6cyclictest12662-21grep21:05:252
26459991911,5cyclictest6965-21apt19:40:102
26459991911,5cyclictest6965-21apt19:40:102
2645999190,17cyclictest1854-21ssh23:47:402
2645899194,12cyclictest1032-21ssh00:35:131
2645899193,9cyclictest8290-21apt-get21:30:091
2645899192,12cyclictest6907-21apt-get20:55:111
2645899192,12cyclictest21815-21apt-get23:15:031
26458991912,4cyclictest30160-21ls22:10:001
26458991912,4cyclictest30160-21ls22:10:001
2645899191,16cyclictest4795-21head23:25:231
26458991911,6cyclictest3458-21needreboot20:45:191
26458991911,6cyclictest22082-21sed20:15:001
26458991911,6cyclictest13333-21cut00:20:161
26458991911,6cyclictest13333-21cut00:20:151
26458991911,6cyclictest1325-21awk22:35:271
26458991911,6cyclictest1213-21sed20:40:181
26458991911,6cyclictest11954-21load21:05:181
26458991911,6cyclictest10818-21sed23:30:191
26458991911,5cyclictest25795-21grep20:20:251
26458991910,7cyclictest20279-1kworker/u9:022:10:231
26458991910,6cyclictest14020-21tune2fs21:10:151
2645899190,9cyclictest1697-21expr23:00:141
2645799197,10cyclictest25396-21fschecks_time00:05:150
2645799192,12cyclictest30753-21apt-get20:35:150
26457991914,3cyclictest2426-21grep19:29:590
26457991913,4cyclictest9075-21/usr/sbin/munin19:45:180
26457991913,4cyclictest32164-21cpuspeed_turbos22:35:130
26457991913,4cyclictest25498-21sh22:05:250
26457991913,4cyclictest25498-21sh22:05:240
26457991913,4cyclictest13038-21sh21:33:220
26457991912,5cyclictest23456-21ls21:40:230
26457991912,5cyclictest14943-21seq23:57:150
26457991911,6cyclictest7088-21cat00:15:200
26457991911,6cyclictest28790-21expr20:30:100
26457991911,6cyclictest26429-21seq20:25:030
26457991911,6cyclictest24589-21cut00:05:000
26457991911,6cyclictest21577-21cat22:50:240
26457991911,6cyclictest13902-21cat19:55:120
26457991911,6cyclictest11955-21expr19:50:150
26457991910,7cyclictest30376-21seq21:45:250
270672180,2sleep10-21swapper/120:25:151
2645999189,6cyclictest28784-21ls20:30:102
2645999189,6cyclictest27444-21pmu-power20:25:202
2645999182,11cyclictest9100-21apt-get19:45:102
2645999181,7cyclictest141rcu_preempt19:30:132
26459991813,3cyclictest7078-21cstates20:55:142
2645899188,7cyclictest19607-21ssh00:25:121
2645899182,11cyclictest11705-21apt-get22:20:121
2645899181,8cyclictest27117-21/usr/sbin/munin21:20:161
2645899181,6cyclictest636-21runrttasks22:53:031
26458991812,4cyclictest29844-21threads20:30:221
26458991811,4cyclictest15325-21sh22:45:461
2645899181,10cyclictest487-21chrt19:25:131
26458991810,6cyclictest16055-21cat20:00:141
26458991810,4cyclictest26713-21forks00:30:141
2645799189,6cyclictest11620-21timerwakeupswit21:55:250
2645799182,11cyclictest21361-21apt-get23:15:040
26457991813,3cyclictest17559-21sh21:36:080
26457991813,3cyclictest17394-21sh00:23:150
26457991813,3cyclictest17394-21sh00:23:150
26457991813,3cyclictest1204-21sh23:00:010
26457991813,3cyclictest11336-21sh23:30:230
26457991812,4cyclictest20000-21/usr/sbin/munin20:10:140
26457991812,4cyclictest14244-21sh21:10:180
26457991811,5cyclictest4833-21cat23:50:130
26457991811,5cyclictest31976-21awk19:20:230
26457991811,5cyclictest26406-21cat20:25:000
26457991811,5cyclictest12-21ksoftirqd/022:45:090
26457991810,6cyclictest6867-21cut20:55:010
26457991810,6cyclictest27582-21systemctl00:30:210
26457991810,6cyclictest19934-21threads22:25:240
26457991810,6cyclictest11539-21diskstats21:05:140
258392180,2chrt0-21swapper/023:41:030
26459991712,3cyclictest28600-21munin-setwarn19:15:012
26459991711,4cyclictest28-21ksoftirqd/219:50:012
2645999171,12cyclictest20025-21apt-get20:10:132
26459991710,5cyclictest26055-21seq20:23:112
2645899179,6cyclictest5793-21cut22:15:171
2645899171,7cyclictest11425-21/usr/sbin/munin19:50:171
26458991712,3cyclictest13564-21/usr/sbin/munin19:55:201
26458991710,5cyclictest29380-21sed19:15:191
2645799179,6cyclictest5243-21perl19:35:140
26457991712,3cyclictest422-21/usr/sbin/munin20:40:130
26457991711,4cyclictest26965-21sh21:20:010
26457991710,5cyclictest18183-21expr20:05:140
2645999162,10cyclictest22163-21apt-get20:15:112
26459991611,3cyclictest17861-21/usr/sbin/munin20:05:132
2645899169,5cyclictest11029-21ls19:50:001
2645799168,5cyclictest9835-21missed_timers21:00:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional