You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 12:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Tue Sep 01, 2026 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
77193823420,2sleep3575-21kworker/3:2+events20:02:373
10149282580,3chrt0-21swapper/200:07:362
718109994941,5cyclictest1043615-21latency_hist00:40:011
9023192480,2sleep3471rcuc/322:12:393
718112994843,3cyclictest708414-21kworker/u8:0+events_unbound19:15:012
718109994438,4cyclictest824472-21kworker/u8:3+events_unbound21:05:011
718109994437,5cyclictest998629-21/usr/sbin/munin23:52:391
718109994335,6cyclictest275-21systemd-journal21:07:401
718107994333,8cyclictest931609-21kworker/u8:1+events_unbound22:57:440
718112994238,3cyclictest760046-21kworker/u8:3+events_unbound19:50:002
718112994137,3cyclictest817635-21kworker/u8:2+events_unbound20:47:492
718112994036,3cyclictest931609-21kworker/u8:1+flush-8:023:42:422
718107993918,9cyclictest943106-21ntpq22:52:430
718109993831,5cyclictest3304708-21kworker/u8:2+flush-8:019:22:421
718109993831,5cyclictest275-21systemd-journal22:51:401
71810799389,13cyclictest765497-21apt-get19:57:280
71810799388,18cyclictest884890-21diskstats21:57:320
718107993833,4cyclictest844686-21kworker/u8:0+events_unbound22:37:490
718107993829,7cyclictest883426-21ntpq21:52:400
718107993816,20cyclictest1-21systemd00:07:390
718107993810,15cyclictest933411-21tail22:42:490
718112993732,4cyclictest894356-21kworker/u8:2+events_unbound22:42:362
718112993732,4cyclictest799339-21kworker/u8:1+events_unbound20:42:442
718112993732,4cyclictest3304708-21kworker/u8:2+events_unbound20:22:392
71810799375,22cyclictest969140-21grep23:22:310
71810799375,22cyclictest969140-21grep23:22:300
718107993726,9cyclictest275-21systemd-journal23:40:010
718107993710,15cyclictest868893-21python321:37:470
7134062373,9sleep10-21swapper/119:06:261
718112993632,3cyclictest844686-21kworker/u8:0+events_unbound21:17:432
718112993632,3cyclictest3304708-21kworker/u8:2+events_unbound20:27:492
71810799365,12cyclictest775631-21df_inode20:07:330
718112993531,3cyclictest931609-21kworker/u8:1+flush-8:022:54:402
718112993531,3cyclictest894356-21kworker/u8:2+flush-8:023:07:452
718112993527,6cyclictest978464-21cat23:30:002
718109993529,4cyclictest967660-21strings23:17:391
718109993529,4cyclictest967660-21strings23:17:391
71810799359,15cyclictest977774-21ntpq23:27:420
71810799359,15cyclictest838786-21ntpq21:07:440
71810799358,16cyclictest815418-21diskstats20:47:330
71810799358,15cyclictest808742-21ntpq20:37:400
71810799358,15cyclictest730587-21diskstats19:22:320
71810799356,14cyclictest848641-21ntpq21:17:400
718107993514,9cyclictest828690-21memory20:57:390
718107993510,13cyclictest788981-21ntpq20:17:410
718107993510,12cyclictest919276-21apt-get22:32:280
718107993510,12cyclictest899948-21/usr/sbin/munin22:12:360
718107993510,12cyclictest893900-21/usr/sbin/munin22:02:500
718112993430,3cyclictest744221-21kworker/u8:1+events_unbound19:32:472
718112993430,3cyclictest744221-21kworker/u8:1+events_unbound19:32:472
718112993429,4cyclictest967844-21kworker/u8:3+flush-8:023:18:392
718112993429,4cyclictest967844-21kworker/u8:3+flush-8:023:18:392
718109993430,3cyclictest1038676-21awk00:35:011
71810799349,14cyclictest952941-21ntpq23:02:410
71810799349,14cyclictest908260-21ntpq22:17:440
71810799349,13cyclictest914339-21/usr/sbin/munin22:27:280
71810799349,12cyclictest1037899-21ntpq00:32:410
71810799349,11cyclictest833870-21ntpq21:02:470
71810799348,15cyclictest774268-21ntpq20:02:440
71810799348,15cyclictest1022847-21ntpq00:17:370
71810799348,15cyclictest1007312-21ntpq23:57:400
71810799348,13cyclictest989012-21diskstats23:42:320
71810799347,16cyclictest839943-21aten2_r5power_c21:12:300
71810799347,16cyclictest1029617-21hddtemp_smartct00:27:320
71810799347,15cyclictest894904-21hddtemp_smartct22:07:310
71810799346,16cyclictest1002220-21memory23:52:390
71810799343,20cyclictest979273-21hddtemp_smartct23:32:330
718107993425,6cyclictest275-21systemd-journal21:22:390
718107993415,9cyclictest761124-21/usr/sbin/munin19:52:350
718107993413,11cyclictest727517-21/usr/sbin/munin19:17:390
718107993410,11cyclictest793962-21ntpq20:22:420
718107993410,10cyclictest968033-21python323:17:460
718107993410,10cyclictest968033-21python323:17:460
718109993328,3cyclictest974492-21irqcore23:27:341
718109993328,3cyclictest742605-21kthreadcore19:32:391
718109993328,3cyclictest742605-21kthreadcore19:32:391
718109993326,5cyclictest755142-21latency_hist19:45:001
718109993326,5cyclictest755142-21latency_hist19:44:591
718109993325,6cyclictest799339-21kworker/u8:1+flush-8:020:32:431
718109993325,5cyclictest844432-21latency_hist21:15:011
71810799339,13cyclictest858783-21/usr/sbin/munin21:27:460
71810799339,13cyclictest1039504-21/usr/sbin/munin00:37:340
71810799339,12cyclictest993978-21/usr/sbin/munin23:47:320
71810799339,12cyclictest803891-21ntpq20:32:420
71810799338,15cyclictest860833-21irqstats21:32:370
71810799338,12cyclictest1012325-21ntpq00:02:420
71810799337,15cyclictest954565-21/usr/sbin/munin23:07:330
71810799337,15cyclictest934586-21diskstats22:47:310
71810799337,15cyclictest909634-21diskstats22:22:330
71810799336,16cyclictest870080-21df_inode21:42:300
71810799335,7cyclictest1018311-21taskset00:10:430
71810799335,15cyclictest722423-21sendmail_mailst19:12:470
71810799335,13cyclictest798844-21/usr/sbin/munin20:27:410
71810799334,17cyclictest744466-21python319:32:480
71810799334,17cyclictest744466-21python319:32:480
718107993311,12cyclictest1024830-21/usr/sbin/munin00:22:340
718107993310,10cyclictest814075-21ntpq20:42:440
718109993225,5cyclictest978895-21apt-get23:32:271
718109993225,5cyclictest967844-21kworker/u8:3+flush-8:023:42:441
718109993225,5cyclictest739776-21latency_hist19:30:001
718109993222,7cyclictest817635-21kworker/u8:2+flush-8:021:27:451
71810799328,10cyclictest784103-21ntpq20:12:420
71810799327,14cyclictest875654-21irqstats21:47:350
71810799327,14cyclictest755819-21df19:47:320
71810799327,14cyclictest735533-21df19:27:310
71810799326,15cyclictest962884-21ntp_states23:12:430
71810799326,15cyclictest962884-21ntp_states23:12:430
718107993210,11cyclictest754395-21ntpq19:42:420
718107993210,11cyclictest754395-21ntpq19:42:420
718107993210,10cyclictest823867-21ntpq20:52:410
718112993127,3cyclictest857456-21kworker/u8:1+events_unbound21:42:432
718112993125,4cyclictest739880-21gpgv19:30:002
718109993125,5cyclictest708414-21kworker/u8:0+flush-8:020:07:341
718109993123,6cyclictest275-21systemd-journal22:16:391
718109993123,5cyclictest933778-21store22:45:001
718107993118,9cyclictest717959-21latency_hist19:10:000
718107993110,11cyclictest747553-21/usr/sbin/munin19:37:370
718112993025,4cyclictest3304708-21kworker/u8:2+events_unbound20:19:392
718112993025,3cyclictest854097-21unixbench_multi21:22:482
718112993025,3cyclictest718484-21aten2_r5power_c19:12:312
718109993025,4cyclictest824472-21kworker/u8:3+events_unbound20:55:011
718112992924,4cyclictest857456-21kworker/u8:1+flush-8:021:47:512
718109992926,2cyclictest918863-21systemd-run22:30:001
718109992924,3cyclictest797294-21kthreadcore20:27:381
718109992923,4cyclictest844686-21kworker/u8:0+events_unbound21:59:591
718109992923,4cyclictest275-21systemd-journal20:55:391
718109992922,5cyclictest275-21systemd-journal22:48:401
718112992822,4cyclictest1033459-21unixbench_multi00:27:482
718109992825,2cyclictest750089-21apt-key19:40:001
718109992822,4cyclictest864223-21unixbench_multi21:32:491
718109992822,4cyclictest809685-21latency_hist20:40:011
718109992821,5cyclictest874745-21/usr/sbin/munin21:47:391
718109992821,5cyclictest760046-21kworker/u8:3+flush-8:020:02:461
718109992820,6cyclictest912811-21sshd22:22:391
718112992722,4cyclictest20823-21cron23:40:002
718109992722,4cyclictest809577-21kworker/u8:0+events_unbound20:50:011
718109992721,4cyclictest953696-21latency_hist23:05:011
718109992721,4cyclictest275-21systemd-journal22:57:391
718109992720,5cyclictest1023649-21basename00:20:001
718109992719,6cyclictest275-21systemd-journal22:40:001
718112992621,3cyclictest967844-21kworker/u8:3+events_unbound23:20:012
718112992621,3cyclictest967844-21kworker/u8:3+events_unbound23:20:012
718112992621,3cyclictest708414-21kworker/u8:0+events_unbound20:10:032
718112992621,3cyclictest3304708-21kworker/u8:2+events_unbound19:20:072
718109992619,5cyclictest879705-21apt-get21:52:271
718109992619,5cyclictest275-21systemd-journal19:50:381
718112992522,2cyclictest825158-21aten2_r5power_p20:57:302
718112992521,3cyclictest744221-21kworker/u8:1+events_unbound19:37:392
718112992521,3cyclictest1036785-21kworker/u8:3+flush-8:000:33:182
71811299251,22cyclictest889888-21cut22:02:342
718109992521,2cyclictest0-21swapper/123:22:291
718109992521,2cyclictest0-21swapper/123:22:291
718109992520,3cyclictest850072-21tr21:22:311
718109992519,5cyclictest857456-21kworker/u8:1+flush-8:022:07:411
718109992519,4cyclictest993722-21apt-get23:47:271
718109992518,5cyclictest785260-21apt-get20:17:291
718109992518,5cyclictest275-21systemd-journal21:45:001
718109992517,5cyclictest275-21systemd-journal00:30:001
718112992420,2cyclictest959281-21cpuspeed_turbos23:12:312
718112992420,2cyclictest959281-21cpuspeed_turbos23:12:312
718112992419,4cyclictest844686-21kworker/u8:0+flush-8:023:32:532
718112992417,6cyclictest751525-21kthreadcore19:42:372
718112992417,6cyclictest751525-21kthreadcore19:42:372
718109992420,2cyclictest0-21swapper/123:12:331
718109992420,2cyclictest0-21swapper/123:12:331
718109992419,3cyclictest923774-21idleruntime-cro22:35:001
718109992419,3cyclictest0-21swapper/100:22:331
718109992419,2cyclictest844-21mta-sts-daemon22:02:471
718109992416,6cyclictest1013486-21/usr/sbin/munin00:07:391
718112992311,9cyclictest1023823-21cat00:20:012
718109992320,2cyclictest1008721-21cpuspeed00:02:311
718109992318,3cyclictest766025-21irqcore19:57:351
718109992318,3cyclictest727523-21kthreadcore19:17:391
718109992318,3cyclictest0-21swapper/120:22:331
71811299226,8cyclictest0-21swapper/221:37:422
71811299224,2cyclictest161rcu_preempt23:02:382
71811299223,8cyclictest161rcu_preempt19:52:352
718112992218,3cyclictest931609-21kworker/u8:1+events_unbound23:47:492
718112992218,2cyclictest805101-21aten2_r5power_e20:37:292
718112992218,2cyclictest765611-21aten2_r5power_v19:57:302
718109992218,3cyclictest846912-21kthreadcore21:17:361
718109992217,3cyclictest958740-21latency_hist23:10:001
718109992217,3cyclictest1008092-21munin-run00:00:001
718112992118,2cyclictest880019-21fschecks_time21:52:322
718112992117,3cyclictest824472-21kworker/u8:3+events_unbound20:53:042
718112992117,3cyclictest0-21swapper/221:30:002
718112992116,4cyclictest1036785-21kworker/u8:3+events_unbound00:35:022
718112992116,3cyclictest0-21swapper/222:15:002
718109992117,2cyclictest0-21swapper/120:12:481
718109992117,2cyclictest0-21swapper/119:12:471
718109992116,3cyclictest0-21swapper/123:37:291
718109992115,4cyclictest0-21swapper/120:42:461
7178672214,10sleep20-21swapper/219:09:082
71811299207,2cyclictest421rcuog/200:02:242
71811299203,9cyclictest840321-21sort21:12:342
718112992015,3cyclictest770535-21aten2_r5power_e20:02:312
71810999209,3cyclictest0-21swapper/119:45:521
718109992012,6cyclictest901672-21tr22:12:391
71811299199,8cyclictest936291-21tr22:47:362
71811299198,3cyclictest161rcu_preempt23:52:322
71811299195,4cyclictest0-21swapper/222:07:462
71811299192,12cyclictest924211-21apt-get22:37:302
71811299191,8cyclictest1028044-21sshd00:22:402
718112991915,3cyclictest931609-21kworker/u8:1+flush-8:023:56:062
718112991914,4cyclictest857456-21kworker/u8:1+flush-8:022:20:372
718112991913,4cyclictest779664-21unixbench_singl20:07:532
718112991912,5cyclictest905628-21tr22:17:362
718112991911,2cyclictest421rcuog/222:57:352
718109991915,3cyclictest868842-21mailstats21:37:461
71811299185,7cyclictest0-21swapper/221:07:492
71811299182,8cyclictest0-21swapper/221:32:442
71811299182,4cyclictest0-21swapper/220:31:082
718112991814,2cyclictest914690-21fschecks_time22:27:332
718112991814,2cyclictest0-21swapper/200:15:002
71811299180,16cyclictest831317-21kthreadcore21:02:382
71811299173,9cyclictest884638-21apt-get21:57:292
71811299171,3cyclictest161rcu_preempt22:32:342
718109991713,2cyclictest0-21swapper/100:15:001
7133112173,8sleep30-21swapper/319:05:113
9232852167,6sleep30-21swapper/322:32:473
8258012166,6sleep30-21swapper/320:57:363
8394712142,8sleep30-21swapper/321:10:003
7454432140,7sleep30-21swapper/319:37:293
7184552142,8sleep30-21swapper/319:12:303
9592682132,7sleep30-21swapper/323:12:323
9592682132,7sleep30-21swapper/323:12:323
9541982132,8sleep30-21swapper/323:07:303
8548032131,8sleep30-21swapper/321:27:303
8455212133,5sleep30-21swapper/321:17:353
8007222133,6sleep30-21swapper/320:32:363
7954972130,2sleep3471rcuc/320:27:343
7840492133,6sleep30-21swapper/320:12:423
7662692132,7sleep30-21swapper/319:57:373
7397572131,8sleep30-21swapper/319:30:003
10236822132,6sleep30-21swapper/300:20:023
9934532122,6sleep30-21swapper/323:45:023
9881872122,6sleep30-21swapper/323:38:403
9734792122,6sleep30-21swapper/323:24:403
9734792122,6sleep30-21swapper/323:24:403
9684892122,6sleep30-21swapper/323:19:463
9684892122,6sleep30-21swapper/323:19:463
9487272120,8sleep30-21swapper/323:00:003
9345812122,6sleep30-21swapper/322:47:323
9334732122,6sleep30-21swapper/322:42:503
9047142122,6sleep30-21swapper/322:17:343
8947772122,6sleep30-21swapper/322:07:313
8881422122,6sleep30-21swapper/321:57:403
8197042122,5sleep30-21swapper/320:50:023
8146572122,6sleep30-21swapper/320:44:163
8094772122,6sleep30-21swapper/320:39:383
7758102122,6sleep30-21swapper/320:07:363
7592962122,6sleep30-21swapper/319:47:423
7402412122,6sleep30-21swapper/319:30:123
7402412122,6sleep30-21swapper/319:30:123
7346612122,6sleep30-21swapper/319:22:503
7257632122,6sleep30-21swapper/319:17:363
10388952122,5sleep30-21swapper/300:35:023
10380282122,6sleep30-21swapper/300:32:463
10118702122,6sleep30-21swapper/300:02:403
9984452112,6sleep30-21swapper/323:50:043
9902032112,5sleep30-21swapper/323:42:383
9786472112,5sleep30-21swapper/323:30:013
9738792112,6sleep30-21swapper/323:26:443
9491042113,5sleep30-21swapper/323:01:363
9437042112,6sleep30-21swapper/322:53:533
9241852112,5sleep30-21swapper/322:37:283
9145892112,5sleep30-21swapper/322:27:333
9098212111,6sleep30-21swapper/322:22:363
8900602112,6sleep30-21swapper/322:02:373
8792852112,5sleep30-21swapper/321:50:013
8746432113,5sleep30-21swapper/321:46:083
8696692112,6sleep30-21swapper/321:40:383
8647102112,6sleep30-21swapper/321:35:163
8598572112,5sleep30-21swapper/321:31:393
8497142112,6sleep30-21swapper/321:21:083
8408442112,5sleep30-21swapper/321:12:373
8299862112,6sleep30-21swapper/321:01:583
8149632112,5sleep30-21swapper/320:45:083
7900142112,6sleep30-21swapper/320:20:123
7850442112,5sleep30-21swapper/320:15:023
7610192112,5sleep30-21swapper/319:52:353
7505392112,5sleep30-21swapper/319:42:003
7505392112,5sleep30-21swapper/319:42:003
10336292112,6sleep30-21swapper/300:28:443
10199762112,5sleep30-21swapper/300:17:323
10186402112,6sleep30-21swapper/300:15:003
10135152112,6sleep30-21swapper/300:07:303
10031432112,5sleep30-21swapper/323:55:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional