You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 05:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Dec 05, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8110210087,9sleep30-21swapper/319:08:213
796729582,9sleep10-21swapper/119:06:321
822828363,16sleep20-21swapper/219:09:542
811928168,8sleep00-21swapper/019:08:270
23282380,0sleep00-21swapper/000:20:170
832799251,23cyclictest0-21swapper/200:30:152
831899250,2cyclictest0-21swapper/020:35:150
124302250,0sleep10-21swapper/123:39:071
832999241,20cyclictest0-21swapper/323:19:023
8323992424,0cyclictest0-21swapper/123:30:021
8327992317,5cyclictest62050irq/128-enp2s0-22:23:482
8327992317,5cyclictest61750irq/125-enp2s0-22:47:502
8327992317,5cyclictest61750irq/125-enp2s0-22:47:492
8327992219,2cyclictest62050irq/128-enp2s0-21:17:532
8327992219,2cyclictest61750irq/125-enp2s0-22:10:442
832799221,21cyclictest0-21swapper/221:50:192
8329992121,0cyclictest0-21swapper/323:04:383
8329992119,1cyclictest62050irq/128-enp2s0-22:48:053
8329992119,1cyclictest62050irq/128-enp2s0-22:48:043
832999210,20cyclictest0-21swapper/320:35:143
8327992119,1cyclictest61950irq/127-enp2s0-21:47:102
8327992118,3cyclictest62050irq/128-enp2s0-21:40:482
832799210,19cyclictest0-21swapper/223:05:532
8329992018,1cyclictest61950irq/127-enp2s0-22:19:143
8329992017,2cyclictest62050irq/128-enp2s0-22:31:473
832999200,1cyclictest0-21swapper/321:45:183
8327992019,1cyclictest61850irq/126-enp2s0-22:17:042
8327992018,1cyclictest61950irq/127-enp2s0-23:22:002
8327992018,1cyclictest61750irq/125-enp2s0-22:03:092
8327992017,2cyclictest62050irq/128-enp2s0-23:48:562
8327992014,1cyclictest62050irq/128-enp2s0-23:52:312
8323992019,1cyclictest0-21swapper/121:11:141
831899200,0cyclictest0-21swapper/023:51:090
8329991917,1cyclictest62050irq/128-enp2s0-22:39:063
8329991917,1cyclictest61950irq/127-enp2s0-22:14:323
8329991916,2cyclictest61750irq/125-enp2s0-21:43:303
832999190,2cyclictest0-21swapper/300:11:523
832799192,3cyclictest9866-21strings00:00:232
8327991918,1cyclictest62050irq/128-enp2s0-23:18:222
8327991918,1cyclictest61850irq/126-enp2s0-22:09:432
8327991917,2cyclictest61750irq/125-enp2s0-00:07:232
8327991917,1cyclictest61950irq/127-enp2s0-23:25:352
8327991916,2cyclictest61850irq/126-enp2s0-23:43:202
8327991913,4cyclictest26082-21apache221:56:002
832399190,17cyclictest0-21swapper/100:30:101
8327991817,1cyclictest61750irq/125-enp2s0-23:04:182
8327991816,2cyclictest62050irq/128-enp2s0-23:36:072
8327991816,2cyclictest62050irq/128-enp2s0-23:33:472
831899182,8cyclictest1703-21awk00:20:010
8329991716,1cyclictest61950irq/127-enp2s0-22:21:033
8329991712,4cyclictest7744-21snmpd21:55:453
8327991716,1cyclictest61850irq/126-enp2s0-00:19:162
8327991714,2cyclictest61950irq/127-enp2s0-22:31:172
8327991714,2cyclictest61950irq/127-enp2s0-21:30:052
8327991714,2cyclictest61850irq/126-enp2s0-00:23:282
832799170,10cyclictest37850irq/131-snd_hda20:00:152
832399170,0cyclictest0-21swapper/123:10:491
832999164,8cyclictest23376-21ssh21:25:193
8329991615,1cyclictest61850irq/126-enp2s0-00:02:193
8327991615,1cyclictest61950irq/127-enp2s0-22:40:412
8327991615,1cyclictest61950irq/127-enp2s0-21:20:552
8327991614,2cyclictest61750irq/125-enp2s0-22:28:382
8327991614,1cyclictest61950irq/127-enp2s0-21:38:322
8327991614,1cyclictest61950irq/127-enp2s0-00:36:202
8323991614,2cyclictest25-21ksoftirqd/122:16:251
831899160,16cyclictest0-21swapper/023:58:020
831899160,12cyclictest0-21swapper/022:10:160
8329991512,2cyclictest61750irq/125-enp2s0-21:21:463
8318991513,1cyclictest0-21swapper/022:02:420
8329991411,2cyclictest20215-21grep22:55:243
8329991411,2cyclictest11973-21wc20:25:013
832999140,13cyclictest0-21swapper/323:54:053
832999140,13cyclictest0-21swapper/300:37:463
8327991412,1cyclictest62050irq/128-enp2s0-00:25:522
8327991412,1cyclictest61750irq/125-enp2s0-22:53:162
8327991411,2cyclictest22843-21tail20:45:172
832399143,6cyclictest0-21swapper/122:25:151
832399142,10cyclictest9022-21tail20:15:281
8323991414,0cyclictest0-21swapper/100:12:191
8323991412,1cyclictest0-21swapper/100:02:051
8323991411,2cyclictest719-21munin-run21:55:001
8323991410,3cyclictest20780-21basename23:45:171
8323991410,2cyclictest21878-21sed19:35:211
8323991410,2cyclictest12655-21grep19:15:261
8318991411,2cyclictest14303-21wc23:15:210
8318991410,3cyclictest7744-21snmpd00:26:160
831899140,12cyclictest23021-21grep21:25:170
832999139,3cyclictest15996-21cat20:30:263
832999139,3cyclictest10957-21tr22:25:173
8329991312,1cyclictest0-21swapper/321:13:023
832999131,10cyclictest0-21swapper/321:35:413
8329991310,2cyclictest9316-21grep19:10:183
8329991310,2cyclictest61950irq/127-enp2s0-22:09:543
8329991310,2cyclictest30958-21grep21:30:233
8329991310,2cyclictest1271-21sort20:00:213
832799139,3cyclictest1835-21grep23:55:162
832799131,5cyclictest14488-21latency_hist20:30:002
8327991311,1cyclictest61850irq/126-enp2s0-21:25:362
8327991310,2cyclictest30719-21cron22:40:002
832799130,2cyclictest0-21swapper/219:15:132
832799130,12cyclictest0-21swapper/221:07:502
832399139,3cyclictest31428-21cron21:05:011
832399139,3cyclictest16589-21sed20:30:341
832399131,8cyclictest6733-21unixbench_singl21:35:251
8323991311,1cyclictest1392-21diskmemload21:44:051
8323991310,2cyclictest25749-21wc20:50:251
8323991310,2cyclictest20584-21systemd-cgroups00:09:241
8323991310,2cyclictest19115-21tr22:55:181
8323991310,2cyclictest12615-21tr22:50:151
8323991310,2cyclictest10704-21wc19:15:001
832399130,1cyclictest141rcu_preempt22:31:261
831899139,2cyclictest30607-21cut22:15:150
831899131,2cyclictest141rcu_preempt21:58:250
8318991310,2cyclictest8808-21grep19:10:140
8318991310,2cyclictest7800-21tr23:10:250
8318991310,2cyclictest28216-21grep23:25:300
8318991310,2cyclictest27238-21ssh21:50:170
8318991310,2cyclictest16444-21grep19:25:170
8318991310,2cyclictest15701-21grep00:05:180
8318991310,2cyclictest13038-21tune2fs20:25:180
8318991310,2cyclictest12055-21grep22:25:220
8318991310,2cyclictest11562-21cron21:39:590
832999129,2cyclictest7945-21cron00:00:003
832999129,2cyclictest7603-21tr20:15:153
832999129,2cyclictest31026-21grep21:00:313
832999129,2cyclictest30998-21grep19:55:193
832999129,2cyclictest27614-21cat21:50:183
832999129,2cyclictest27180-21grep20:55:173
832999129,2cyclictest27180-21grep20:55:163
832999129,2cyclictest22418-21cat23:45:283
832999129,2cyclictest15655-21grep23:40:263
832999129,2cyclictest1409-21basename21:10:013
832999129,2cyclictest12908-21df22:50:163
832999127,3cyclictest18307-21apt-get19:30:133
832999122,9cyclictest0-21swapper/323:23:273
8329991211,1cyclictest61950irq/127-enp2s0-00:27:113
832799129,2cyclictest28131-21tr19:50:142
832799129,2cyclictest26205-21cut19:45:182
832799129,2cyclictest20563-21cat22:55:292
832799127,4cyclictest12491-21ssh23:14:432
832799121,11cyclictest1392-21diskmemload21:12:452
8327991210,1cyclictest0-21swapper/200:10:282
832399129,3cyclictest245392sleep119:40:261
832399129,2cyclictest8857-21grep22:00:151
832399129,2cyclictest6709-21grep20:10:281
832399129,2cyclictest5420-21ssh23:09:201
832399129,2cyclictest27274-21if_err_enp2s023:25:221
832399129,2cyclictest24752-21grep21:25:291
832399129,2cyclictest2087-21cut23:55:181
832399129,2cyclictest17989-21cat19:30:011
832399129,2cyclictest16160-21cut22:05:171
832399129,2cyclictest1381-21sh21:10:001
832399128,3cyclictest30978-21grep19:55:181
832399128,3cyclictest26098-21apache222:49:341
832399128,3cyclictest26098-21apache222:49:331
832399128,3cyclictest25-21ksoftirqd/122:35:441
832399128,3cyclictest23534-21run-parts00:40:001
832399120,2cyclictest0-21swapper/123:40:561
831899129,2cyclictest32548-21cat19:59:590
831899129,2cyclictest28051-21tr20:55:230
831899129,2cyclictest28051-21tr20:55:220
831899129,2cyclictest18449-21awk22:55:000
831899129,2cyclictest1387-21apt21:10:000
831899128,3cyclictest8136-21sed20:15:180
831899128,3cyclictest21013-21chrt23:20:220
831899126,2cyclictest224-21systemd-journal23:05:000
8318991210,1cyclictest9012-21grep23:35:290
831899120,11cyclictest6703-21sed20:10:290
832999119,1cyclictest7283-21grep23:10:223
832999119,1cyclictest26269-21grep19:45:193
832999119,1cyclictest20442-21grep20:40:193
832999118,2cyclictest7050-21ssh23:35:123
832999118,2cyclictest3961-21ssh00:20:323
832999118,2cyclictest28840-21cut00:15:173
832999118,2cyclictest27412-21cut23:25:223
832999118,2cyclictest22297-21grep19:35:263
832999118,2cyclictest2038-21sh23:05:403
832999118,2cyclictest14993-21basename00:05:113
832999118,2cyclictest14813-21sh00:30:493
832999118,2cyclictest11975-21ssh21:16:573
832999112,4cyclictest224-21systemd-journal22:40:003
832999112,3cyclictest0-21swapper/323:33:113
8329991111,0cyclictest0-21swapper/322:00:263
832999110,10cyclictest12929-21taskset19:18:323
832799119,1cyclictest289442sleep220:59:002
832799119,1cyclictest289442sleep220:58:592
832799118,2cyclictest5334-21seq20:10:162
832799118,2cyclictest32404-21awk20:00:002
832799118,2cyclictest25063-21df_inode20:50:182
832799118,2cyclictest22469-21chrt19:36:012
832799118,2cyclictest17968-21grep20:35:172
832399118,2cyclictest8010-21grep00:25:231
832399118,2cyclictest66942sleep100:24:501
832399118,2cyclictest31544-21ssh23:52:481
832399118,2cyclictest30431-21sh23:03:301
832399118,2cyclictest28669-21sh22:14:041
832399118,2cyclictest28429-21tr00:15:161
832399118,2cyclictest28178-21tr19:50:151
832399118,2cyclictest21309-21wc20:40:291
832399118,2cyclictest20435-21grep21:45:231
832399118,2cyclictest20244-21rm21:23:001
832399118,2cyclictest17939-21sort20:35:181
832399118,2cyclictest1522-21taskset22:41:161
832399118,2cyclictest1339-21ssh21:55:101
832399118,2cyclictest13150-21cut20:25:181
832399112,1cyclictest30103-21apache_processe21:30:181
831899118,2cyclictest9909-21cut20:20:150
831899118,2cyclictest8529-21sh21:15:110
831899118,2cyclictest6251-21grep22:45:190
831899118,2cyclictest6251-21grep22:45:180
831899118,2cyclictest4723-21cron20:09:590
831899118,2cyclictest32282-21sh23:05:140
831899118,2cyclictest30191-21cat19:55:000
831899118,2cyclictest26617-21ssh22:35:540
831899118,2cyclictest26169-21grep19:45:180
831899118,2cyclictest25244-21expr20:50:200
831899118,2cyclictest24171-21ssh21:48:190
831899118,2cyclictest22581-21cat20:45:160
831899118,2cyclictest22476-21sort00:10:190
831899118,2cyclictest22277-21cat23:45:280
831899118,2cyclictest22062-21ssh22:56:490
831899118,2cyclictest21471-21expr19:35:180
831899118,2cyclictest18726-21ssh22:30:260
831899118,2cyclictest18294-21ssh00:35:070
831899118,2cyclictest16710-21tail22:05:190
831899118,2cyclictest1624-21sh21:32:260
831899118,2cyclictest15368-21ssh00:31:310
831899118,2cyclictest14081-21rm00:04:250
831899118,2cyclictest14049-21seq21:40:290
831899118,2cyclictest14017-21grep23:40:140
831899118,2cyclictest1321-21grep23:30:190
831899118,2cyclictest10298-21ssh22:24:540
831899117,3cyclictest24295-21ls19:40:220
831899110,10cyclictest0-21swapper/021:10:360
832999108,1cyclictest43772sleep320:05:363
832999108,1cyclictest25027-21taskset19:43:023
832999108,1cyclictest24136-21chrt20:49:023
832999108,1cyclictest12849-21expr20:25:153
832999107,2cyclictest7100-21cron20:14:593
832999107,2cyclictest26204-21grep20:50:303
832999107,2cyclictest17568-21taskset19:26:533
832999107,2cyclictest14669-21run-parts19:20:223
832799108,2cyclictest2702-21awk20:05:162
832799108,1cyclictest23196-21taskset19:40:142
832799108,1cyclictest204022sleep219:34:562
832799108,1cyclictest16181-21expr19:25:142
832799108,1cyclictest13617-21cut19:20:132
832799108,1cyclictest10254-21sed19:10:262
832799107,2cyclictest30953-21sed21:00:322
832799107,2cyclictest16058-21tail20:30:262
832799107,2cyclictest11383-21ls20:20:282
832799106,3cyclictest21409-21grep20:40:292
832399108,1cyclictest27132-21sed19:45:271
832399108,1cyclictest24062-21taskset20:48:061
832399108,1cyclictest19250-21cut19:30:181
832399108,1cyclictest1069-21basename20:00:191
832399107,2cyclictest9799-21ssh21:15:211
832399107,2cyclictest4619-21taskset20:08:411
832399107,2cyclictest19661-21cut23:20:141
832399107,2cyclictest11897-21apt-get20:24:591
832399101,7cyclictest0-21swapper/121:00:011
832399101,7cyclictest0-21swapper/121:00:001
832399101,5cyclictest4902-21cut22:20:181
832399100,4cyclictest0-21swapper/123:15:161
831899108,1cyclictest930-21tune2fs20:00:190
831899108,1cyclictest29634-21cat21:00:210
831899108,1cyclictest16098-21ntpq20:30:260
831899108,1cyclictest15469-21chrt21:20:050
831899108,1cyclictest13661-21expr19:20:130
831899108,1cyclictest11286-21expr19:15:150
831899107,2cyclictest31766-21df22:40:190
831899107,2cyclictest21609-21taskset20:42:090
831899107,2cyclictest18845-21df_abs19:30:150
83299997,1cyclictest28837-21cut19:50:183
83279997,1cyclictest7331-21chrt20:15:042
83239997,1cyclictest15125-21chrt19:21:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional