You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 13:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 08, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
273182238153,21sleep20-21swapper/219:09:532
271522235161,25sleep00-21swapper/019:07:420
270912230163,55sleep30-21swapper/319:06:563
272772201165,24sleep10-21swapper/119:09:201
1665121420,5sleep32742799cyclictest23:35:253
274229910299,2cyclictest0-21swapper/222:16:222
274229910198,2cyclictest0-21swapper/220:22:012
27422998985,3cyclictest0-21swapper/221:06:322
27422998783,3cyclictest0-21swapper/219:28:342
256622590,3sleep025666-21unixbench-2d23:55:280
221852540,2sleep10-21swapper/122:35:181
310822530,2sleep30-21swapper/321:40:263
74142500,3sleep00-21swapper/023:15:180
27410994111,20cyclictest30807-21/usr/sbin/munin00:10:160
2741099409,12cyclictest19286-21if_enp1s021:15:160
2741099376,8cyclictest8293-21ntpq00:30:200
27410993715,13cyclictest808-21snmpd22:46:230
27410993712,10cyclictest30650-21grep21:40:230
2741099369,14cyclictest3461-21kworker/u8:122:00:240
27410993615,13cyclictest2386-21kworker/u8:200:35:150
27410993613,12cyclictest15856-21open_files22:20:210
27410993611,8cyclictest10647-21kworker/0:123:35:170
27410993610,11cyclictest7704-21cpuspeed_turbos20:50:130
2741099359,15cyclictest0-21swapper/022:40:010
27410993517,7cyclictest16128-21diskmemload22:50:150
27410993513,13cyclictest13190-21/usr/sbin/munin23:30:130
27410993512,15cyclictest7919-1kworker/u9:022:35:580
27410993414,13cyclictest27684-21df_inode21:35:130
27410993414,11cyclictest26718-21/usr/sbin/munin20:20:210
27410993413,12cyclictest14369-21taskset22:18:000
2741099339,13cyclictest10913-21fschecks_time22:10:170
2741099336,12cyclictest7323-21munin-run20:50:010
27410993315,6cyclictest13079-21cat23:30:020
27410993315,10cyclictest808-21snmpd20:30:590
27410993313,7cyclictest16128-21diskmemload21:30:010
27410993313,17cyclictest16349-21/usr/sbin/munin21:10:140
27410993312,14cyclictest17297-21df22:25:110
27410993311,9cyclictest4505-21crond19:30:010
27427993219,3cyclictest0-21swapper/321:59:493
2741099329,14cyclictest0-21swapper/021:20:190
2741099329,10cyclictest29666-21ntp_states00:05:220
2741099329,10cyclictest29666-21ntp_states00:05:210
27410993217,11cyclictest22106-21awk20:10:000
27410993214,6cyclictest30176-21egrep22:55:120
27410993214,6cyclictest30176-21egrep22:55:120
27410993213,11cyclictest14154-21/usr/sbin/munin21:05:230
27410993211,9cyclictest32620-21latency21:45:190
27410993211,9cyclictest32620-21latency21:45:190
27410993211,12cyclictest21975-21/usr/sbin/munin23:50:170
2741799311,11cyclictest0-21swapper/121:50:161
27410993117,5cyclictest10201-21ntp_states19:40:210
27410993113,6cyclictest5456-21ntp_offset23:10:210
27410993112,6cyclictest32708-21/usr/sbin/munin19:20:240
27410993112,16cyclictest18892-21wc23:40:240
27410993110,13cyclictest8179-21/usr/sbin/munin22:05:200
2741799301,19cyclictest20112-21missed_timers22:30:221
2741099309,8cyclictest22327-21/usr/sbin/munin20:10:150
2741099309,12cyclictest17921-21/usr/sbin/munin20:00:150
2741099308,10cyclictest909-21sssd_nss22:30:210
2741099307,13cyclictest0-21swapper/019:30:130
2741099307,10cyclictest0-21swapper/019:35:220
27410993023,4cyclictest29597-21hddtemp_smartct20:25:150
27410993014,5cyclictest29086-21sed19:10:260
27410993012,10cyclictest808-21snmpd23:01:260
27410993011,7cyclictest4179-21perf23:10:010
27410993011,6cyclictest10615-21chrt23:23:170
27427992917,2cyclictest23151-21wc23:50:203
2742299295,3cyclictest0-21swapper/219:59:592
2741799294,8cyclictest0-21swapper/100:05:201
2741799294,8cyclictest0-21swapper/100:05:201
2741799292,18cyclictest19256-40dnf23:41:251
2741799291,19cyclictest30188-21gltestperf21:40:161
2741799291,18cyclictest0-21swapper/123:00:231
2741799291,17cyclictest0-21swapper/121:56:121
2741099299,7cyclictest20128-21sh23:45:140
2741099298,11cyclictest0-21swapper/021:50:220
2741099297,6cyclictest544-21/usr/sbin/munin00:15:190
27410992920,6cyclictest28373-21/usr/sbin/munin00:05:010
27410992920,5cyclictest10094-21diskstats20:55:140
27410992919,7cyclictest14871-21/usr/sbin/munin19:50:240
27410992911,15cyclictest16717-21grep19:55:200
27410992910,8cyclictest827-21runrttasks21:56:510
2742799280,25cyclictest31567-21if_enp2s000:10:173
2741799281,4cyclictest0-21swapper/123:20:431
2741799281,20cyclictest87550irq/128-enp1s0-21:12:081
2741799281,16cyclictest0-21swapper/122:26:521
2741799281,12cyclictest0-21swapper/122:01:201
27410992820,5cyclictest5950-21memory00:25:190
27410992819,6cyclictest932-21/usr/sbin/munin20:35:230
27410992819,6cyclictest932-21/usr/sbin/munin20:35:230
27410992819,6cyclictest2844-21/usr/sbin/munin00:20:170
27410992819,6cyclictest24994-21/usr/sbin/munin20:15:170
27410992811,6cyclictest12622-21hddtemp_smartct21:00:150
27410992811,10cyclictest25720-21perl21:30:160
2741799274,7cyclictest0-21swapper/123:57:211
2741799273,16cyclictest20335-21sshd21:17:511
2741799272,8cyclictest0-21swapper/122:19:131
2741799271,8cyclictest0-21swapper/123:07:501
2741799271,18cyclictest0-21swapper/100:28:321
2741799271,12cyclictest0-21swapper/123:39:001
2741799271,12cyclictest0-21swapper/122:48:071
2741799271,11cyclictest0-21swapper/122:12:031
2741799271,11cyclictest0-21swapper/121:49:591
2741799271,11cyclictest0-21swapper/121:49:591
2741799271,11cyclictest0-21swapper/100:13:431
27410992719,5cyclictest31323-21irqstats19:15:200
27410992719,5cyclictest12005-21hddtemp_smartct19:45:170
2742799261,23cyclictest0-21swapper/323:20:203
2742799261,23cyclictest0-21swapper/300:08:163
2742799261,23cyclictest0-21swapper/300:08:163
2742799261,22cyclictest28254-21memory21:35:183
2741799261,9cyclictest0-21swapper/123:30:461
2741799261,11cyclictest20638-21kworker/u8:121:25:161
2741799261,11cyclictest0-21swapper/121:39:171
2741799261,10cyclictest0-21swapper/100:32:121
2741799261,10cyclictest0-21swapper/100:19:101
27410992619,4cyclictest3146-40dnf20:40:070
97482250,1sleep30-21swapper/319:40:153
36952250,2sleep30-21swapper/323:05:263
2742799251,22cyclictest0-21swapper/300:25:213
2742799251,21cyclictest16128-21diskmemload21:24:533
2742799251,21cyclictest15076-21sshd23:34:113
2742799251,21cyclictest0-21swapper/321:26:213
2742799251,21cyclictest0-21swapper/300:24:073
2741799255,11cyclictest0-21swapper/123:16:001
2741799253,16cyclictest0-21swapper/121:34:461
2741799252,17cyclictest0-21swapper/123:10:321
27417992512,3cyclictest87550irq/128-enp1s0-22:54:531
2741799251,17cyclictest0-21swapper/122:05:181
2741799251,16cyclictest0-21swapper/122:55:161
2741799251,16cyclictest0-21swapper/122:55:161
2741799251,15cyclictest0-21swapper/122:43:411
2742799248,8cyclictest6781-21irqstats22:00:173
2742799241,5cyclictest0-21swapper/323:16:053
2742799241,5cyclictest0-21swapper/322:43:033
2742799241,21cyclictest31719-21gltestperf20:30:153
2742799241,21cyclictest0-21swapper/322:45:133
2742799241,21cyclictest0-21swapper/322:27:143
2742799241,21cyclictest0-21swapper/322:21:133
2742799241,21cyclictest0-21swapper/320:11:183
2742799241,20cyclictest996-21netstat23:00:203
2742799241,20cyclictest932-21/usr/sbin/munin20:35:193
2742799241,20cyclictest932-21/usr/sbin/munin20:35:193
2742799241,20cyclictest39-21rcuc/323:10:233
2742799241,20cyclictest26053-21memory21:30:203
2742799241,20cyclictest25580-21ntpq20:15:233
2742799241,20cyclictest12570-21sshd22:15:113
2742299241,20cyclictest9138-21df_inode23:20:132
2741799248,9cyclictest0-21swapper/120:50:121
27417992412,3cyclictest0-21swapper/123:26:261
2741799241,19cyclictest29268-21diskstats20:25:131
2741799241,17cyclictest0-21swapper/100:22:151
2741799241,14cyclictest0-21swapper/100:03:591
2741799241,12cyclictest0-21swapper/123:49:171
2742799233,9cyclictest87350irq/126-enp1s0-22:35:213
2742799233,4cyclictest0-21swapper/323:42:113
2742799231,9cyclictest0-21swapper/323:45:173
2742799231,9cyclictest0-21swapper/322:50:153
2742799231,20cyclictest0-21swapper/323:29:043
2742799231,20cyclictest0-21swapper/322:11:403
2742799231,20cyclictest0-21swapper/320:50:173
2742799231,19cyclictest808-21snmpd19:15:453
2742799231,15cyclictest0-21swapper/321:54:543
27427992311,2cyclictest13854-21cstates19:50:123
2742799231,11cyclictest11164-21sshd00:37:113
2742799231,10cyclictest0-21swapper/323:58:023
2741799231,16cyclictest0-21swapper/119:11:371
27417992311,3cyclictest24298-21munin-run20:15:011
2741799231,11cyclictest25464-21missed_timers20:15:211
2741799230,17cyclictest2386-21kworker/u8:200:35:131
2742799221,6cyclictest0-21swapper/321:45:253
2742799221,6cyclictest0-21swapper/321:45:253
27427992215,4cyclictest27050-21sed00:00:153
2742799221,1cyclictest0-21swapper/321:12:543
2742799221,19cyclictest401ktimersoftd/319:32:243
2742799221,19cyclictest401ktimersoftd/319:23:183
2742799221,19cyclictest0-21swapper/321:16:363
2742799221,10cyclictest20294-21/usr/sbin/munin22:30:243
27427992210,2cyclictest8179-21/usr/sbin/munin22:05:203
2742799220,20cyclictest0-21swapper/320:45:123
2742799220,20cyclictest0-21swapper/300:30:233
2742299221,5cyclictest20031-21unixbench-2d21:15:262
2741799228,3cyclictest12913-21missed_timers21:00:191
2741799225,10cyclictest0-21swapper/123:50:461
2741799221,15cyclictest0-21swapper/120:00:121
2741799221,15cyclictest0-21swapper/119:30:241
2741799221,12cyclictest7100-21tr19:35:121
2741799221,11cyclictest9138-21/usr/sbin/munin19:40:161
2741799221,11cyclictest790-21in:imjournal20:35:011
2741799221,11cyclictest14842-21sort22:20:111
2741799220,15cyclictest23670-21kworker/u8:021:20:081
2742799211,19cyclictest401ktimersoftd/320:44:053
27427992111,2cyclictest0-21swapper/320:05:253
2742799211,11cyclictest0-21swapper/319:59:123
2742799211,10cyclictest0-21swapper/300:15:203
2742799210,19cyclictest0-21swapper/320:00:113
2742299211,7cyclictest32530-21cat19:20:012
2742299211,7cyclictest0-21swapper/200:06:432
2742299211,7cyclictest0-21swapper/200:06:432
2742299211,7cyclictest0-21swapper/200:02:162
2742299211,6cyclictest17510-21proc_pri21:10:202
2741799217,8cyclictest21-21rcuc/120:40:211
2741799216,9cyclictest0-21swapper/121:05:251
2741799216,9cyclictest0-21swapper/119:27:021
2741799216,9cyclictest0-21swapper/119:15:451
2741799211,10cyclictest932-21/usr/sbin/munin20:35:131
2741799211,10cyclictest932-21/usr/sbin/munin20:35:131
2741799210,11cyclictest0-21swapper/120:20:161
2742799201,6cyclictest0-21swapper/319:29:443
2742799201,3cyclictest0-21swapper/319:10:253
2742799201,17cyclictest0-21swapper/321:05:213
2742799201,10cyclictest0-21swapper/322:55:353
2742799201,10cyclictest0-21swapper/322:55:353
2742299205,3cyclictest23620-21sh21:25:162
2742299204,8cyclictest19662-21sshd20:02:072
2742299202,6cyclictest20128-21/usr/sbin/munin20:05:172
2742299201,7cyclictest0-21swapper/222:35:142
2742299201,7cyclictest0-21swapper/200:30:252
2742299201,7cyclictest0-21swapper/200:26:322
2742299201,6cyclictest31337-21sshd21:42:572
2742299201,6cyclictest0-21swapper/223:31:312
2742299201,6cyclictest0-21swapper/223:09:312
2742299201,6cyclictest0-21swapper/222:30:272
2742299201,5cyclictest23796-21sshd23:53:112
2742299201,5cyclictest0-21swapper/221:46:102
2742299201,5cyclictest0-21swapper/221:46:102
2742299201,5cyclictest0-21swapper/219:11:542
27422992013,4cyclictest4389-21grep21:55:152
2742299201,12cyclictest0-21swapper/220:53:402
2742299200,6cyclictest0-21swapper/220:15:152
2741799205,9cyclictest0-21swapper/119:50:201
2741799201,9cyclictest5352-21/usr/sbin/munin20:45:241
27417992012,3cyclictest0-21swapper/120:09:181
2741799201,14cyclictest0-21swapper/119:55:561
2741799201,14cyclictest0-21swapper/119:22:241
2741799201,13cyclictest0-21swapper/120:56:031
2741799201,11cyclictest12921-21users19:45:271
67152190,7sleep2311ktimersoftd/223:15:122
294722190,4sleep230-21rcuc/221:40:012
2742799196,3cyclictest87650irq/129-enp1s0-21:00:113
2742299192,4cyclictest10531-21awk00:35:242
2742299191,7cyclictest0-21swapper/223:57:102
2742299191,7cyclictest0-21swapper/222:40:282
2742299191,6cyclictest1179-21processes23:00:242
2742299191,6cyclictest0-21swapper/223:40:152
2742299191,6cyclictest0-21swapper/222:27:332
2742299191,6cyclictest0-21swapper/222:00:252
2742299191,6cyclictest0-21swapper/220:55:462
2742299191,6cyclictest0-21swapper/219:42:252
2742299191,6cyclictest0-21swapper/200:10:262
2742299191,5cyclictest0-21swapper/223:46:242
2742299191,5cyclictest0-21swapper/223:27:532
2742299191,5cyclictest0-21swapper/219:35:482
27422991914,3cyclictest31780-21cut20:30:152
2742299191,15cyclictest5056-21gltestperf23:10:162
2742299191,10cyclictest0-21swapper/221:33:212
2742799182,15cyclictest401ktimersoftd/319:45:253
2742799181,5cyclictest0-21swapper/320:55:453
2742799181,3cyclictest26709-21sshd20:20:073
2742799181,15cyclictest0-21swapper/319:35:123
2742799181,14cyclictest29555-21gltestperf20:25:153
2742299185,3cyclictest0-21swapper/219:54:172
2742299183,3cyclictest87650irq/129-enp1s0-19:20:122
2742299182,3cyclictest3146-40dnf20:40:082
2742299181,6cyclictest0-21swapper/222:50:172
2742299181,6cyclictest0-21swapper/221:51:432
2742299181,5cyclictest1706-21ntp_states00:15:212
2742299181,5cyclictest10379-21/usr/sbin/munin22:10:152
2742299181,5cyclictest0-21swapper/223:35:102
2742299181,5cyclictest0-21swapper/222:20:202
2742299181,4cyclictest0-21swapper/222:08:042
2742299181,4cyclictest0-21swapper/221:22:442
27422991813,3cyclictest11674-21df_inode19:45:152
2742299171,8cyclictest0-21swapper/220:49:122
2742299171,6cyclictest32-21ksoftirqd/222:56:182
2742299171,6cyclictest32-21ksoftirqd/222:56:182
2742299171,4cyclictest26090-21df22:45:132
27422991711,3cyclictest4751-21acpi19:30:122
27422991711,3cyclictest22699-21fschecks_count20:10:142
2742299170,5cyclictest0-21swapper/200:21:042
2742299161,4cyclictest0-21swapper/220:25:192
2742299161,14cyclictest0-21swapper/221:01:212
2742299157,5cyclictest0-21swapper/220:35:212
2742299157,5cyclictest0-21swapper/220:35:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional