You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 23:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 09, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1870729984,10sleep00-21swapper/007:05:280
1903429467,8sleep30-21swapper/307:09:343
1882028474,6sleep10-21swapper/107:06:461
1899628363,9sleep20-21swapper/207:09:052
19163992727,0cyclictest0-21swapper/311:14:273
1915899260,3cyclictest0-21swapper/109:49:231
1916399241,20cyclictest0-21swapper/311:08:063
1916199240,1cyclictest141rcu_preempt10:59:492
1915899240,24cyclictest0-21swapper/110:17:461
1916399220,1cyclictest11882-21kworker/3:209:37:593
1916199220,21cyclictest0-21swapper/211:42:552
19158992218,2cyclictest0-21swapper/111:34:071
19158992112,9cyclictest25-21ksoftirqd/109:10:151
1915899210,20cyclictest0-21swapper/110:23:421
1915899200,15cyclictest0-21swapper/110:39:131
264212190,0sleep00-21swapper/011:14:490
1916399190,0cyclictest0-21swapper/309:47:213
19158991916,2cyclictest0-21swapper/110:34:351
19158991915,3cyclictest13309-21latency_hist12:05:001
1915899191,14cyclictest0-21swapper/108:50:161
1915899190,18cyclictest0-21swapper/112:20:141
1915899190,18cyclictest0-21swapper/109:53:371
1915699196,6cyclictest30349-21sendmail-msp09:40:010
1915699194,8cyclictest7143-21awk08:59:590
1916199180,1cyclictest0-21swapper/212:08:452
1916199180,14cyclictest0-21swapper/207:50:132
1916199180,14cyclictest0-21swapper/207:50:122
19163991716,1cyclictest11945-21diskmemload10:22:233
19163991715,1cyclictest2232-21ssh12:36:183
1916399170,2cyclictest0-21swapper/309:24:583
1916199170,8cyclictest225-21systemd-journal09:18:032
1916199170,13cyclictest0-21swapper/209:56:082
1915899175,11cyclictest11945-21diskmemload10:27:351
1915899173,9cyclictest0-21swapper/111:05:121
19158991717,0cyclictest0-21swapper/112:30:111
1915899170,17cyclictest0-21swapper/110:11:331
1915699171,6cyclictest0-21swapper/007:43:330
19156991713,2cyclictest537-21snmpd10:23:160
1916399163,7cyclictest225-21systemd-journal11:00:013
19163991615,1cyclictest0-21swapper/311:43:093
1916399160,3cyclictest0-21swapper/312:05:553
1916399160,1cyclictest0-21swapper/312:29:123
1916199161,1cyclictest16158-21users09:30:262
1916199160,15cyclictest0-21swapper/211:32:002
1915899160,15cyclictest0-21swapper/111:55:131
1915699164,8cyclictest7784-21ssh10:05:130
1915699162,10cyclictest2133-21ssh11:00:140
1915699160,9cyclictest0-21swapper/007:55:110
1916399154,1cyclictest0-21swapper/311:37:523
1916399153,11cyclictest28410-21rm12:33:063
1916399150,1cyclictest0-21swapper/311:21:423
19161991513,1cyclictest1481-21apache_volume07:40:142
1915899159,5cyclictest22814-21grep09:35:151
19158991514,1cyclictest0-21swapper/112:35:351
1916399143,3cyclictest2395-21rm10:01:113
19163991411,2cyclictest724-21sed07:35:253
19163991411,2cyclictest7100-21grep12:20:173
19163991411,2cyclictest12346-21ssh10:26:403
1916399140,3cyclictest0-21swapper/311:47:303
1916399140,12cyclictest1687-21sed08:45:263
1916199148,3cyclictest11040-21diskstats11:05:142
19161991411,2cyclictest2604-21grep11:20:012
19161991411,2cyclictest2604-21grep11:20:002
19161991410,3cyclictest32193-21grep12:15:262
19161991410,3cyclictest25803-21cat10:55:002
1916199140,13cyclictest0-21swapper/212:11:482
1915899141,8cyclictest0-21swapper/109:05:131
19158991411,1cyclictest0-21swapper/110:59:111
19158991410,3cyclictest26834-21wc08:30:221
19158991410,3cyclictest15522-21cron08:10:001
1915899140,2cyclictest0-21swapper/111:00:541
1915899140,1cyclictest0-21swapper/110:50:231
1915699143,6cyclictest8336-21munin-run07:55:010
1915699143,6cyclictest8336-21munin-run07:55:000
1915699141,6cyclictest17924-21awk08:15:000
19156991414,0cyclictest0-21swapper/007:45:130
19156991410,3cyclictest5636-21cut12:00:140
19156991410,3cyclictest22730-21wc07:15:180
1915699140,13cyclictest15603-21wc09:30:210
1915699140,12cyclictest15909-21tail09:50:170
1916399139,3cyclictest21740-21sed11:30:273
1916399133,5cyclictest225-21systemd-journal07:20:003
19163991310,2cyclictest9519-21munin-run09:04:593
19163991310,2cyclictest5956-21munin-plugin-st07:50:003
19163991310,2cyclictest18942-21grep08:15:173
19163991310,2cyclictest17317-21head10:10:213
1916399130,13cyclictest0-21swapper/309:54:033
1916399130,11cyclictest0-21swapper/309:32:083
1916199139,3cyclictest9946-21expr10:45:152
1916199139,3cyclictest7044-21grep09:45:142
1916199139,3cyclictest32202-21grep08:44:592
1916199135,3cyclictest225-21systemd-journal09:05:002
19161991312,1cyclictest0-21swapper/211:23:302
19161991311,1cyclictest0-21swapper/209:36:252
19161991310,2cyclictest6221-21ls08:55:232
19161991310,2cyclictest20040-21cat11:10:222
19161991310,2cyclictest19561-21sh12:27:422
19161991310,2cyclictest13151-21cron08:05:012
19161991310,2cyclictest11945-21diskmemload10:12:452
1916199130,2cyclictest141rcu_preempt10:00:412
1916199130,2cyclictest0-21swapper/209:24:482
1916199130,13cyclictest0-21swapper/210:44:492
1916199130,12cyclictest32112-21df12:35:162
1916199130,12cyclictest0-21swapper/211:00:542
1915899139,3cyclictest10266-21tail07:55:241
1915899139,2cyclictest3578-21cron07:45:011
1915899135,2cyclictest225-21systemd-journal11:20:091
1915899131,1cyclictest4945-21apt-get09:25:021
19158991310,2cyclictest24019-21grep12:10:251
19158991310,2cyclictest22609-21sort12:30:011
19158991310,2cyclictest22049-21tr07:15:131
1915699139,3cyclictest5892-21grep09:25:140
1915699139,3cyclictest27994-21wc10:55:260
1915699139,3cyclictest26372-21grep10:15:260
1915699139,2cyclictest8941-21wc09:00:260
1915699138,3cyclictest32731-21tr08:45:140
1915699136,1cyclictest225-21systemd-journal09:15:000
1915699132,7cyclictest11945-21diskmemload12:15:200
1915699131,3cyclictest141rcu_preempt11:08:460
1915699131,3cyclictest0-21swapper/011:20:140
19156991310,2cyclictest21031-21basename09:15:130
19156991310,2cyclictest17118-21sh10:10:190
19156991310,2cyclictest15717-21wc12:25:180
1916399129,2cyclictest9651-21ssh10:45:153
1916399129,2cyclictest8858-21grep07:55:123
1916399129,2cyclictest5604-21grep08:55:173
1916399129,2cyclictest27437-21munin-plugin-st08:35:003
1916399129,2cyclictest27284-21grep10:35:263
1916399129,2cyclictest27036-21grep07:25:143
1916399129,2cyclictest25708-21cron10:54:593
1916399129,2cyclictest22927-21grep12:10:153
1916399129,2cyclictest12799-21cut09:10:153
1916399129,2cyclictest11234-21tr08:00:163
1916399128,3cyclictest14686-21wc08:05:193
1916399120,2cyclictest0-21swapper/311:01:503
1916399120,2cyclictest0-21swapper/309:59:543
1916399120,0cyclictest0-21swapper/312:00:273
1916399120,0cyclictest0-21swapper/310:40:453
1916199129,2cyclictest6192-21grep12:00:172
1916199129,2cyclictest32026-21cat09:40:202
1916199129,2cyclictest30904-21wc11:55:202
1916199129,2cyclictest30156-21cat07:30:192
1916199129,2cyclictest24663-21tr07:20:142
1916199129,2cyclictest22995-21grep07:15:212
1916199129,2cyclictest13182-21wc11:25:262
1916199129,2cyclictest11936-21sh09:09:592
1916199122,9cyclictest55550irq/126-enp2s0-09:50:122
19161991211,1cyclictest55650irq/127-enp2s0-10:36:392
1916199120,11cyclictest18751-21cat10:30:242
1915899129,2cyclictest8660-21wc09:00:221
1915899129,2cyclictest32308-21wc08:44:591
1915899129,2cyclictest31772-21wc12:15:221
1915899129,2cyclictest29323-21sed08:35:271
1915899129,2cyclictest15756-21grep12:05:261
1915899129,2cyclictest11815-21sed08:00:191
1915899129,2cyclictest0-21swapper/111:40:201
1915899128,3cyclictest6403-21chrt10:04:211
1915899127,4cyclictest21551-21sed08:20:171
1915899123,5cyclictest225-21systemd-journal11:20:011
1915899123,5cyclictest225-21systemd-journal11:20:001
1915899122,9cyclictest11945-21diskmemload11:47:001
19158991211,1cyclictest0-21swapper/110:41:181
1915899120,1cyclictest11945-21diskmemload09:21:241
1915699129,2cyclictest32632-21tail10:00:170
1915699129,2cyclictest31689-21grep09:40:170
1915699129,2cyclictest29117-21tail11:15:270
1915699129,2cyclictest29117-21tail11:15:260
1915699129,2cyclictest28643-21tr11:35:140
1915699129,2cyclictest27804-21grep08:35:150
1915699129,2cyclictest23735-21tr09:55:130
1915699129,2cyclictest21963-21grep11:50:160
1915699129,2cyclictest21277-21wc11:30:230
1915699129,2cyclictest20806-21grep08:20:130
1915699129,2cyclictest15553-21basename08:10:010
1915699129,2cyclictest14489-21sort12:05:160
1915699129,2cyclictest11343-21wc09:05:250
1915699128,3cyclictest17033-21ssh10:30:110
1915699128,2cyclictest18433-21tr08:15:160
1915699125,4cyclictest5971-21gzip11:40:230
1915699121,3cyclictest4737-21ssh12:38:150
19156991210,1cyclictest3196-21grep10:40:260
1916399119,1cyclictest25049-21entropy10:15:153
1916399119,1cyclictest21803-21grep08:20:213
1916399119,1cyclictest18324-21cut10:30:193
1916399118,2cyclictest6524-21hwlatdetect09:25:163
1916399118,2cyclictest504-21ssh09:40:313
1916399118,2cyclictest3417-21taskset07:43:003
1916399118,2cyclictest32075-21taskset08:43:323
1916399118,2cyclictest31608-21ntp_kernel_err12:15:203
1916399118,2cyclictest29837-21head07:30:163
1916399118,2cyclictest29285-21cat08:35:253
1916399118,2cyclictest25152-21head07:20:163
1916399118,2cyclictest24597-21ssh09:16:413
1916399118,2cyclictest23291-21ssh11:50:313
1916399118,2cyclictest23205-21grep08:25:133
1916399118,2cyclictest14760-21ssh11:26:273
1916399118,2cyclictest12084-21cat09:10:003
19163991110,1cyclictest0-21swapper/311:15:143
19163991110,1cyclictest0-21swapper/311:15:133
1916399110,2cyclictest0-21swapper/310:05:213
1916399110,10cyclictest29253-21sh11:55:013
1916199118,2cyclictest9435-21ssh10:25:172
1916199118,2cyclictest6282-21sh12:20:132
1916199118,2cyclictest5917-21tr09:25:142
1916199118,2cyclictest5649-21chrt07:46:102
1916199118,2cyclictest3760-21ssh10:21:332
1916199118,2cyclictest28471-21wget11:35:132
1916199118,2cyclictest24785-21expr10:15:142
1916199118,2cyclictest23777-21cut12:30:152
1916199118,2cyclictest23577-21grep08:25:152
1916199118,2cyclictest13240-21df_abs09:10:172
1916199118,2cyclictest10181-21sh10:05:462
1916199117,3cyclictest26049-21head08:30:142
1916199114,6cyclictest11488-21apache211:45:032
1916199110,10cyclictest28136-21expr08:35:162
1915899118,2cyclictest30449-21wc07:30:221
1915899118,2cyclictest24213-21ssh09:16:211
1915899118,2cyclictest23337-21ssh09:55:071
1915899118,2cyclictest22911-21chrt08:25:041
1915899118,2cyclictest22357-21ssh11:11:401
1915899118,2cyclictest1931-21chrt08:47:161
1915899118,2cyclictest18496-21expr08:15:161
1915899118,2cyclictest16401-21cut08:10:171
1915899118,2cyclictest14806-21rm11:26:281
1915899118,2cyclictest14211-21cut09:30:121
1915899118,2cyclictest11759-21taskset10:06:581
1915899118,2cyclictest10686-21wc10:45:191
1915899117,3cyclictest21512-21perf07:14:591
1915899111,9cyclictest11945-21diskmemload09:40:231
1915899110,10cyclictest31219-21sh11:36:081
1915699118,2cyclictest9605-21basename10:25:180
1915699118,2cyclictest4215-21sshd08:51:080
1915699118,2cyclictest31142-21ssh09:20:220
1915699118,2cyclictest30741-21sort08:40:150
1915699118,2cyclictest29766-21tr11:55:140
1915699118,2cyclictest29384-21cut07:30:140
1915699118,2cyclictest25922-21sed10:35:150
1915699118,2cyclictest25575-21grep08:30:110
1915699118,2cyclictest24829-21taskset08:27:100
1915699118,2cyclictest24392-21cut07:20:130
1915699118,2cyclictest23174-21cut12:30:140
1915699118,2cyclictest17987-21ssh10:50:150
1915699118,2cyclictest14722-21sed11:45:260
1915699118,2cyclictest11334-21ssh11:25:080
1915699118,2cyclictest11221-21grep08:00:160
1915699118,2cyclictest11213-21chrt12:22:450
1915699117,2cyclictest19645-21tr07:10:140
1915699110,10cyclictest17159-21ssh10:49:480
1916399108,1cyclictest6602-21cat07:50:143
1916399108,1cyclictest6602-21cat07:50:143
1916399108,1cyclictest19870-21expr07:10:143
1916399108,1cyclictest17666-21taskset08:12:033
1916399103,1cyclictest0-21swapper/308:50:133
1916199108,1cyclictest225262sleep208:23:052
1916199108,1cyclictest20157-21chrt08:18:232
1916199108,1cyclictest1905-21chrt08:46:572
1916199108,1cyclictest14628-21cut08:05:192
1916199107,2cyclictest31621-21hostname07:35:132
1916199107,2cyclictest2378-21chrt08:50:042
1916199107,2cyclictest19750-21expr07:10:132
1916199101,7cyclictest26985-21cut07:25:132
1916199100,9cyclictest22221-21ssh11:50:182
1915899108,1cyclictest6949-21grep07:50:161
1915899108,1cyclictest6949-21grep07:50:151
1915899108,1cyclictest57862sleep107:47:581
1915899108,1cyclictest28067-21grep07:25:201
1915899107,2cyclictest32071-21sed07:35:141
1915899107,2cyclictest24849-21cat07:20:151
1915899103,4cyclictest21778-21df_abs11:50:141
1915699107,2cyclictest28891-21grep07:30:000
1915699107,2cyclictest22955-21grep12:10:150
1915699103,3cyclictest6744-21ssh09:45:110
1915699100,9cyclictest881-21chrt07:36:190
191619997,1cyclictest15843-21aten2_r7power_e08:10:162
191619993,2cyclictest225-21systemd-journal08:00:002
191589997,1cyclictest6838-21taskset08:56:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional