You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 19:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Fri Aug 29, 2025 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2530352996154,5cyclictest2530478-21kworker/u8:3+flush-8:007:15:461
2530349996157,3cyclictest2633103-21kworker/u8:2+flush-8:009:10:410
2530352995950,6cyclictest2561804-21kworker/u8:1+flush-8:008:00:271
2530349995952,5cyclictest2597063-21kworker/u8:3+events_unbound08:25:410
2530349995853,4cyclictest2551875-21kworker/u8:0+events_unbound07:55:440
2530352995749,5cyclictest2551875-21kworker/u8:0+flush-8:007:50:311
2530352995443,8cyclictest2820141-21ntpq11:55:341
2530349995449,3cyclictest2741665-21kworker/u8:0+events_unbound11:55:390
2530349995446,3cyclictest2692138-21kworker/u8:1+events_unbound10:25:280
2530352995346,5cyclictest2561804-21kworker/u8:1+events_unbound09:05:011
2530352995344,6cyclictest2666869-21latency_hist09:25:011
2530349995349,3cyclictest1133768-21kworker/u8:4+events_unbound07:35:460
2530349995342,3cyclictest2551875-21kworker/u8:0+events_unbound08:00:030
2530349995248,3cyclictest2736453-21kworker/u8:3+events_unbound11:35:270
2530349995242,3cyclictest1133768-21kworker/u8:4+events_unbound07:15:110
2530349995147,3cyclictest2741665-21kworker/u8:0+events_unbound12:10:300
2530349995146,4cyclictest2561804-21kworker/u8:1+events_unbound08:35:350
2530349995146,4cyclictest2561804-21kworker/u8:1+events_unbound08:35:350
2530349995141,3cyclictest2561804-21kworker/u8:1+events_unbound08:15:290
2530349995140,3cyclictest2561804-21kworker/u8:1+events_unbound09:01:520
2530356995036,5cyclictest2736453-21kworker/u8:3+events_unbound10:45:372
2530349995046,3cyclictest2736453-21kworker/u8:3+events_unbound11:05:420
2530349995046,3cyclictest2617267-21kworker/u8:0+events_unbound09:15:460
2530349995046,3cyclictest2561804-21kworker/u8:1+events_unbound09:00:010
2530349995045,4cyclictest2707244-21kworker/u8:2+events_unbound10:20:270
2530349995045,4cyclictest2597063-21kworker/u8:3+events_unbound08:45:460
2530349995040,8cyclictest2786648-21/usr/sbin/munin11:25:410
2530352994943,4cyclictest1133768-21kworker/u8:4+flush-8:007:35:291
2530349994937,9cyclictest2805301-21ntpq11:40:370
2530349994937,9cyclictest2805301-21ntpq11:40:370
2530352994844,3cyclictest2736453-21kworker/u8:3+flush-8:011:35:391
2530349994844,3cyclictest2741665-21kworker/u8:0+events_unbound12:00:540
2530352994741,4cyclictest2652370-21kworker/u8:3+flush-8:009:45:301
2530349994743,3cyclictest2597063-21kworker/u8:3+events_unbound08:30:430
2530356994640,5cyclictest2707244-21kworker/u8:2+flush-8:010:40:362
2530349994642,3cyclictest2551875-21kworker/u8:0+events_unbound07:50:160
253034999460,43cyclictest2535442-21install07:15:010
2530352994436,6cyclictest276-21systemd-journal10:35:001
2530352994436,6cyclictest276-21systemd-journal10:35:001
2530352994432,9cyclictest2640960-21ntpq08:55:371
2530349994440,3cyclictest2736453-21kworker/u8:3+events_unbound11:30:280
2530352994334,6cyclictest2530189-21latency_hist07:10:011
2530352994333,7cyclictest2621106-21ntpq08:35:371
2530352994333,7cyclictest2621106-21ntpq08:35:371
2530349994339,3cyclictest2633103-21kworker/u8:2+events_unbound09:25:360
2530349994339,3cyclictest2633103-21kworker/u8:2+events_unbound09:05:300
2530349994335,3cyclictest2633103-21kworker/u8:2+events_unbound09:45:350
2530356994232,8cyclictest276-21systemd-journal10:27:502
2530352994236,4cyclictest2530226-21kworker/u8:2+flush-8:007:30:431
2530352994236,4cyclictest2530226-21kworker/u8:2+flush-8:007:30:431
2530349994239,2cyclictest2845936-21kworker/u8:0+events_unbound12:28:510
2530349994238,3cyclictest2707244-21kworker/u8:2+events_unbound11:14:520
2530349994238,3cyclictest2681739-21kworker/u8:0+events_unbound09:51:510
2530349994237,4cyclictest2692138-21kworker/u8:1+events_unbound10:13:520
2530356994136,4cyclictest2692138-21kworker/u8:1+flush-8:010:30:382
2530356994136,4cyclictest2692138-21kworker/u8:1+flush-8:010:30:372
2530352994137,3cyclictest2633103-21kworker/u8:2+flush-8:009:35:311
2530352994133,6cyclictest2652370-21kworker/u8:3+flush-8:009:55:321
2530349994137,3cyclictest2796270-21kworker/u8:1+events_unbound12:19:510
2530349994137,3cyclictest2796270-21kworker/u8:1+events_unbound11:45:300
2530349994137,3cyclictest2741665-21kworker/u8:0+events_unbound12:09:520
2530349994137,3cyclictest2707244-21kworker/u8:2+events_unbound10:56:510
2530349994137,3cyclictest2707244-21kworker/u8:2+events_unbound10:52:520
2530349994137,3cyclictest2633103-21kworker/u8:2+events_unbound09:40:300
2530349994137,3cyclictest2633103-21kworker/u8:2+events_unbound09:40:300
2530349994137,3cyclictest2633103-21kworker/u8:2+events_unbound09:20:170
2530349994136,3cyclictest2796270-21kworker/u8:1+events_unbound11:51:500
2530349994135,4cyclictest2652370-21kworker/u8:3+flush-8:009:55:330
2530356994028,4cyclictest2736453-21kworker/u8:3+events_unbound11:15:422
2530352994031,6cyclictest2746767-21latency_hist10:45:011
2530349994036,3cyclictest2811889-21kworker/u8:2+events_unbound12:31:030
2530349994036,3cyclictest2692138-21kworker/u8:1+events_unbound10:15:370
2530349994036,3cyclictest2681739-21kworker/u8:0+events_unbound10:05:050
2530349994036,3cyclictest2617267-21kworker/u8:0+events_unbound08:52:510
2530349994036,3cyclictest2551875-21kworker/u8:0+events_unbound08:22:520
2530349994032,3cyclictest2561804-21kworker/u8:1+events_unbound07:44:010
2530352993934,3cyclictest2551875-21kworker/u8:0+events_unbound07:40:271
2530349993935,3cyclictest2692138-21kworker/u8:1+events_unbound10:30:430
2530349993935,3cyclictest2692138-21kworker/u8:1+events_unbound10:30:430
2530349993935,3cyclictest2561804-21kworker/u8:1+events_unbound08:11:510
2530349993935,3cyclictest2551875-21kworker/u8:0+events_unbound07:46:510
2530349993935,3cyclictest2530226-21kworker/u8:2+events_unbound07:30:030
2530349993935,3cyclictest2530226-21kworker/u8:2+events_unbound07:30:020
2530356993833,4cyclictest2707244-21kworker/u8:2+flush-8:010:05:322
2530352993832,3cyclictest2633103-21kworker/u8:2+flush-8:009:05:271
2530352993831,5cyclictest2681739-21kworker/u8:0+flush-8:010:00:401
2530349993835,2cyclictest2597063-21kworker/u8:3+events_unbound08:40:370
2530349993834,3cyclictest2796270-21kworker/u8:1+events_unbound12:20:430
2530349993834,3cyclictest2707244-21kworker/u8:2+events_unbound10:42:380
2530349993834,3cyclictest2707244-21kworker/u8:2+events_unbound10:35:280
2530349993834,3cyclictest2652370-21kworker/u8:3+events_unbound09:33:510
2530349993834,3cyclictest2633103-21kworker/u8:2+events_unbound09:35:420
2530349993834,3cyclictest2530226-21kworker/u8:2+events_unbound07:23:510
2530349993833,3cyclictest2838910-21kworker/u8:4+events_unbound12:37:470
2530356993734,2cyclictest2530226-21kworker/u8:2+flush-8:007:10:312
2530352993730,5cyclictest2707244-21kworker/u8:2+flush-8:010:50:361
2530352993729,6cyclictest2582057-21latency_hist08:00:011
2530349993733,3cyclictest2741665-21kworker/u8:0+events_unbound11:24:510
2530349993733,3cyclictest2736453-21kworker/u8:3+events_unbound11:19:510
2530349993733,3cyclictest2736453-21kworker/u8:3+events_unbound11:04:370
2530349993733,3cyclictest2530478-21kworker/u8:3+events_unbound07:26:170
2530352993631,4cyclictest2707244-21kworker/u8:2+flush-8:010:20:311
2530352993627,7cyclictest2622263-21/usr/sbin/munin08:40:431
2530356993530,4cyclictest2561804-21kworker/u8:1+flush-8:008:00:372
2530352993526,6cyclictest2781444-21latency_hist11:20:011
253034999354,2cyclictest161rcu_preempt10:45:360
2530356993429,4cyclictest2681739-21kworker/u8:0+flush-8:010:10:372
2530356993428,4cyclictest2686752-21gpgv09:45:002
2530356993428,4cyclictest2686752-21gpgv09:45:002
2530356993428,4cyclictest2571560-21unixbench_singl07:45:432
2530352993428,4cyclictest2530226-21kworker/u8:2+flush-8:007:25:431
2530352993427,5cyclictest2707244-21kworker/u8:2+events_unbound11:20:401
2530356993327,4cyclictest2652370-21kworker/u8:3+flush-8:009:30:002
2530352993325,6cyclictest2711029-21ntpq10:05:381
2530349993329,3cyclictest2652370-21kworker/u8:3+events_unbound10:01:200
2530349993329,3cyclictest2561804-21kworker/u8:1+events_unbound08:06:070
25300672334,11sleep20-21swapper/207:08:452
2530356993227,4cyclictest2736453-21kworker/u8:3+events_unbound10:55:392
2530352993228,3cyclictest2707244-21kworker/u8:2+flush-8:011:10:471
2530352993227,4cyclictest2796270-21kworker/u8:1+flush-8:012:00:411
2530352993226,4cyclictest2811889-21kworker/u8:2+events_unbound12:25:261
2530352993225,5cyclictest2597063-21kworker/u8:3+flush-8:008:45:441
2530352993224,6cyclictest2692138-21kworker/u8:1+events_unbound10:25:321
2530352993224,6cyclictest2602125-21latency_hist08:20:011
2530356993126,4cyclictest2831493-21kworker/u8:3+flush-8:012:15:252
2530356993126,4cyclictest2681739-21kworker/u8:0+flush-8:009:50:362
2530356993125,4cyclictest2707244-21kworker/u8:2+flush-8:011:20:172
2530352993127,3cyclictest2597063-21kworker/u8:3+events_unbound08:25:331
2530352993124,5cyclictest2811889-21kworker/u8:2+events_unbound12:10:021
2530352993124,4cyclictest2551875-21kworker/u8:0+flush-8:008:25:011
2530352993120,8cyclictest2695961-21ntpq09:50:361
2530356993027,2cyclictest2856306-21cron12:35:002
2530356993026,3cyclictest2845936-21kworker/u8:0+events_unbound12:21:462
2530356993026,3cyclictest2707244-21kworker/u8:2+flush-8:011:15:012
2530356993025,3cyclictest2652370-21kworker/u8:3+flush-8:009:15:012
2530352993026,3cyclictest2750729-21needreboot10:45:351
2530352993025,4cyclictest2551875-21kworker/u8:0+flush-8:007:45:281
2530352993024,5cyclictest2667104-21kworker/u8:1+flush-8:009:25:371
2530352993024,4cyclictest2791336-21/usr/sbin/munin11:30:011
2530352993024,4cyclictest2530478-21kworker/u8:3+flush-8:007:20:251
2530352993022,6cyclictest2838910-21kworker/u8:4+flush-8:012:15:361
2530356992925,3cyclictest2561804-21kworker/u8:1+flush-8:008:50:472
2530356992925,2cyclictest2741779-21sh10:39:592
2530356992924,3cyclictest2817104-21irqrtprio11:55:292
2530356992922,4cyclictest2845936-21kworker/u8:0+flush-8:012:25:062
2530356992915,8cyclictest2835272-21ntpq12:10:382
2530352992925,3cyclictest2530226-21kworker/u8:2+flush-8:007:13:211
2530352992925,2cyclictest0-21swapper/112:40:001
2530352992923,4cyclictest2736453-21kworker/u8:3+flush-8:011:05:011
2530352992923,4cyclictest2652370-21kworker/u8:3+events_unbound09:45:001
2530352992923,4cyclictest2652370-21kworker/u8:3+events_unbound09:45:001
2530356992825,2cyclictest2651765-21cron09:10:002
2530356992824,2cyclictest2612692-21fschecks_time08:30:282
2530352992824,3cyclictest2741665-21kworker/u8:0+events_unbound11:30:371
2530352992824,3cyclictest2707244-21kworker/u8:2+flush-8:010:18:211
2530352992822,4cyclictest2771729-21latency_hist11:10:011
2530356992724,2cyclictest2763420-21kthreadcore11:00:312
2530356992723,2cyclictest2587659-21aten2_r5power_v08:05:252
2530356992722,4cyclictest2561804-21kworker/u8:1+flush-8:008:55:372
2530356992721,3cyclictest2796270-21kworker/u8:1+flush-8:012:05:202
2530352992723,2cyclictest2656712-21cron09:15:001
2530352992722,4cyclictest2633103-21kworker/u8:2+flush-8:009:17:211
2530352992719,6cyclictest276-21systemd-journal09:35:001
253035299270,25cyclictest2712269-21/usr/sbin/munin10:10:221
2530356992622,2cyclictest2621599-21grep08:35:432
2530356992622,2cyclictest2621599-21grep08:35:432
2530356992621,3cyclictest2622508-21cpuspeed_turbos08:40:332
2530352992622,3cyclictest2796270-21kworker/u8:1+flush-8:012:05:311
2530352992620,4cyclictest1-21systemd12:33:501
2530356992521,2cyclictest2687285-21aten2_r5power_p09:45:252
2530356992520,3cyclictest2813588-21kthreadcore11:50:332
253035699250,10cyclictest27900132grep11:25:342
253035299253,2cyclictest0-21swapper/111:45:341
2530352992521,3cyclictest2707244-21kworker/u8:2+flush-8:010:35:521
2530352992520,3cyclictest2761621-21idleruntime-cro11:00:001
2530352992519,5cyclictest2796270-21kworker/u8:1+events_unbound11:40:451
2530352992519,5cyclictest2796270-21kworker/u8:1+events_unbound11:40:441
2530352992518,5cyclictest2838910-21kworker/u8:4+events_unbound12:20:261
2530356992421,2cyclictest0-21swapper/212:37:322
2530356992421,2cyclictest0-21swapper/211:45:132
2530356992421,2cyclictest0-21swapper/211:41:422
2530356992421,2cyclictest0-21swapper/211:41:422
2530356992421,2cyclictest0-21swapper/211:35:082
2530356992421,2cyclictest0-21swapper/211:31:512
2530356992421,2cyclictest0-21swapper/210:22:442
2530356992421,2cyclictest0-21swapper/210:16:252
2530356992421,2cyclictest0-21swapper/209:56:502
2530356992421,2cyclictest0-21swapper/209:36:452
2530356992421,2cyclictest0-21swapper/209:30:122
2530356992421,2cyclictest0-21swapper/208:47:292
2530356992421,2cyclictest0-21swapper/208:21:052
2530356992421,2cyclictest0-21swapper/208:11:142
2530356992421,2cyclictest0-21swapper/207:35:062
2530356992421,2cyclictest0-21swapper/207:31:502
2530356992421,2cyclictest0-21swapper/207:31:492
2530356992421,2cyclictest0-21swapper/207:25:152
2530356992421,2cyclictest0-21swapper/207:21:592
2530356992420,3cyclictest2736453-21kworker/u8:3+flush-8:010:50:032
2530356992420,3cyclictest2561804-21kworker/u8:1+flush-8:009:00:412
2530356992418,4cyclictest2662900-21sed09:20:312
253035699241,3cyclictest2598327-21irqstats08:15:312
253035699240,21cyclictest2656723-21apt-get09:15:152
2530352992421,2cyclictest0-21swapper/111:52:401
2530352992421,2cyclictest0-21swapper/108:51:531
253035299241,3cyclictest2591353-21ntpq08:05:371
253035299240,22cyclictest2614507-21kthreadcore08:30:331
25301172243,14sleep00-21swapper/007:09:230
253035699232,2cyclictest0-21swapper/207:16:172
2530356992320,2cyclictest0-21swapper/212:01:172
2530356992320,2cyclictest0-21swapper/211:08:442
2530356992320,2cyclictest0-21swapper/208:27:242
2530356992320,2cyclictest0-21swapper/207:57:432
2530356992320,2cyclictest0-21swapper/207:51:242
253035699231,20cyclictest2702086-21apt-get10:00:252
2530352992317,5cyclictest2561804-21kworker/u8:1+flush-8:008:10:251
253035699222,2cyclictest0-21swapper/207:40:582
26467482202,15sleep30-21swapper/309:04:383
27337202190,2chrt481ktimers/310:30:323
27337202190,2chrt481ktimers/310:30:323
25300732194,8sleep30-21swapper/307:08:503
28355192188,6sleep30-21swapper/312:10:443
25349402182,4sleep30-21swapper/307:11:033
26408872168,5sleep30-21swapper/308:55:353
25866782160,2sleep3471rcuc/308:00:413
27715282142,7sleep30-21swapper/311:10:013
25768232143,7sleep30-21swapper/307:55:013
25406142140,4sleep3481ktimers/307:15:353
28461872131,8sleep30-21swapper/312:25:003
28201432132,7sleep30-21swapper/311:55:353
27616852132,6sleep30-21swapper/311:00:023
27484382133,6sleep30-21swapper/310:45:333
27219842132,6sleep30-21swapper/310:20:013
26161632132,7sleep30-21swapper/308:30:403
25589502132,7sleep30-21swapper/307:35:353
28560512122,6sleep30-21swapper/312:31:503
28505702122,6sleep30-21swapper/312:25:403
28261522122,6sleep30-21swapper/312:05:023
28118002122,6sleep30-21swapper/311:50:293
27962362122,6sleep30-21swapper/311:35:013
27817392122,6sleep30-21swapper/311:20:243
27807842122,6sleep30-21swapper/311:15:443
27420832122,6sleep30-21swapper/310:40:243
27370592122,6sleep30-21swapper/310:35:263
27077542122,6sleep30-21swapper/310:05:333
26970252122,5sleep30-21swapper/309:55:023
26959302122,6sleep30-21swapper/309:50:363
26857532122,6sleep30-21swapper/309:40:373
26857532122,6sleep30-21swapper/309:40:373
26720732122,6sleep30-21swapper/309:30:183
26668172123,6sleep30-21swapper/309:24:323
26572372122,6sleep30-21swapper/309:15:253
26543512122,6sleep30-21swapper/309:10:323
26319352122,5sleep30-21swapper/308:50:023
26272432122,6sleep30-21swapper/308:45:243
26210672122,7sleep30-21swapper/308:35:383
26210672122,7sleep30-21swapper/308:35:373
25922642122,5sleep30-21swapper/308:10:013
25884322122,6sleep30-21swapper/308:05:323
25776612122,6sleep30-21swapper/307:55:293
25669862122,6sleep30-21swapper/307:45:023
25626112122,6sleep30-21swapper/307:40:283
25513702122,6sleep30-21swapper/307:25:443
25466252122,6sleep30-21swapper/307:20:423
28601882112,5sleep30-21swapper/312:35:353
28408242113,5sleep30-21swapper/312:15:453
28253122113,5sleep30-21swapper/312:00:413
28067122112,5sleep30-21swapper/311:45:283
28015022112,5sleep30-21swapper/311:40:223
28015022112,5sleep30-21swapper/311:40:223
27956112112,5sleep30-21swapper/311:30:473
27865002112,5sleep30-21swapper/311:25:023
27752562112,5sleep30-21swapper/311:10:333
27569182112,5sleep30-21swapper/310:55:243
27524622112,5sleep30-21swapper/310:50:313
27273762112,5sleep30-21swapper/310:25:313
27172732112,5sleep30-21swapper/310:15:283
27121752112,5sleep30-21swapper/310:10:023
27020602112,6sleep30-21swapper/310:00:243
26917732112,5sleep30-21swapper/309:48:363
26773562112,6sleep30-21swapper/309:35:283
26671542112,5sleep30-21swapper/309:25:253
26073242112,5sleep30-21swapper/308:25:023
26024262112,6sleep30-21swapper/308:20:243
26009892112,5sleep30-21swapper/308:15:353
26470172102,4sleep30-21swapper/309:05:023
26223202102,5sleep30-21swapper/308:40:313
25520462102,5sleep30-21swapper/307:30:013
25520462102,5sleep30-21swapper/307:30:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional