You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 03:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Mar 22, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5134210866,38sleep00-21swapper/019:07:500
527628269,9sleep20-21swapper/219:09:412
521828155,24sleep10-21swapper/119:08:561
317228067,9sleep30-21swapper/319:05:083
539599250,24cyclictest0-21swapper/300:09:283
538899240,23cyclictest0-21swapper/122:49:461
538499234,17cyclictest0-21swapper/022:15:130
539399220,22cyclictest0-21swapper/200:09:152
539399220,21cyclictest0-21swapper/223:09:592
5395992119,1cyclictest39-21ksoftirqd/321:23:123
5395992118,2cyclictest39-21ksoftirqd/323:00:003
5395992012,2cyclictest39-21ksoftirqd/300:02:113
539399202,2cyclictest141rcu_preempt21:11:352
5393992020,0cyclictest0-21swapper/221:25:372
539399200,20cyclictest0-21swapper/221:37:222
5388992017,2cyclictest38150irq/131-snd_hda23:45:251
538899200,2cyclictest0-21swapper/100:01:541
538899200,1cyclictest0-21swapper/122:54:361
538899200,19cyclictest0-21swapper/120:30:221
538899191,10cyclictest0-21swapper/122:15:131
538899190,3cyclictest0-21swapper/100:23:541
539399184,11cyclictest0-21swapper/220:00:142
5393991817,1cyclictest0-21swapper/222:55:452
5393991817,1cyclictest0-21swapper/221:49:232
539399180,1cyclictest0-21swapper/222:05:172
538899181,1cyclictest10666-21sort20:25:261
538899180,18cyclictest0-21swapper/121:25:521
538899180,18cyclictest0-21swapper/100:26:001
538899180,0cyclictest0-21swapper/122:04:141
538499182,12cyclictest9678-21grep21:35:180
539599175,2cyclictest141rcu_preempt21:53:013
539399171,15cyclictest0-21swapper/223:49:052
538899172,2cyclictest13195-21sed00:30:191
5388991715,1cyclictest0-21swapper/121:35:101
538499172,1cyclictest141rcu_preempt23:23:020
539599164,10cyclictest27385-21df_abs22:05:153
539599160,2cyclictest0-21swapper/323:12:233
539399165,6cyclictest8444-21apt-get23:30:142
539399163,1cyclictest980750kworker/u8:121:57:292
5393991616,0cyclictest0-21swapper/223:55:102
539399160,16cyclictest0-21swapper/223:37:592
538899162,13cyclictest30592-21diskmemload23:05:391
5388991616,0cyclictest0-21swapper/123:52:421
5388991615,1cyclictest0-21swapper/119:55:181
5388991613,3cyclictest0-21swapper/121:15:341
5388991610,3cyclictest30592-21diskmemload22:58:071
538899160,1cyclictest0-21swapper/121:41:521
538499165,5cyclictest537-21snmpd23:56:440
5384991613,2cyclictest0-21swapper/021:48:060
538499160,1cyclictest0-21swapper/023:45:130
539599158,6cyclictest39-21ksoftirqd/300:10:243
5395991512,1cyclictest141rcu_preempt22:25:253
5395991511,1cyclictest39-21ksoftirqd/321:29:153
539599150,15cyclictest0-21swapper/323:05:493
5393991511,1cyclictest141rcu_preempt22:33:252
538899151,3cyclictest30496-21ssh22:25:291
5388991513,1cyclictest0-21swapper/120:20:131
538899150,0cyclictest0-21swapper/122:33:461
538499153,6cyclictest3179-21munin-run00:25:000
5384991511,2cyclictest6493-21wc23:10:000
538499150,9cyclictest23620-21awk21:25:000
539599145,1cyclictest141rcu_preempt21:00:183
539599143,1cyclictest141rcu_preempt22:18:293
5395991410,3cyclictest12735-21perf19:25:003
539599140,1cyclictest141rcu_preempt23:03:083
539599140,1cyclictest141rcu_preempt21:35:493
539599140,1cyclictest0-21swapper/321:49:343
539399149,5cyclictest23778-21apache223:26:592
5393991410,3cyclictest22226-21cron19:45:002
5393991410,2cyclictest5968-21tr22:50:152
5393991410,2cyclictest28364-21cat21:05:002
539399140,3cyclictest0-21swapper/223:16:502
539399140,12cyclictest16353-21cut20:40:112
538899149,4cyclictest25-21ksoftirqd/121:46:501
538899148,5cyclictest38150irq/131-snd_hda23:58:201
5388991410,3cyclictest14431-21wc19:25:251
538899140,8cyclictest0-21swapper/123:25:241
538899140,1cyclictest0-21swapper/122:42:341
538499146,7cyclictest7754-21munin-setwarn19:15:000
538499143,5cyclictest29237-21perf22:45:000
538499142,6cyclictest19867-21sendmail19:40:010
5384991410,3cyclictest12840-21sed00:30:170
5384991410,2cyclictest15123-21wc19:30:000
538499140,12cyclictest16161-21cat20:40:000
539599139,3cyclictest9003-21cron23:50:003
539599138,4cyclictest39-21ksoftirqd/323:55:133
539599138,2cyclictest39-21ksoftirqd/300:35:003
539599137,5cyclictest39-21ksoftirqd/323:52:323
539599136,6cyclictest39-21ksoftirqd/319:50:133
5395991312,1cyclictest39-21ksoftirqd/300:23:323
5395991311,1cyclictest381rcuc/323:41:353
5395991311,1cyclictest31867-21sort21:10:163
5395991311,1cyclictest30592-21diskmemload23:27:113
5395991310,2cyclictest7938-21grep21:15:153
5395991310,2cyclictest6516-21grep22:50:183
5395991310,2cyclictest2693-21tune2fs20:10:173
5395991310,2cyclictest24791-21grep23:20:173
5395991310,2cyclictest11173-21grep21:55:193
539599130,1cyclictest0-21swapper/323:37:003
539599130,12cyclictest30583-21cron21:10:003
539599130,12cyclictest17030-21grep20:40:143
539399139,3cyclictest3727-21sed22:10:172
539399139,3cyclictest16917-21uname21:20:202
539399133,6cyclictest225-21systemd-journal22:40:002
5393991311,1cyclictest30592-21diskmemload22:23:212
5393991310,2cyclictest7479-21ssh21:15:132
5393991310,2cyclictest28774-21grep22:25:152
5393991310,2cyclictest19968-21cat19:40:002
5393991310,2cyclictest16089-21tail19:30:162
5393991310,2cyclictest11412-21expr00:10:162
539399130,0cyclictest0-21swapper/223:10:172
538899139,3cyclictest8351-21grep19:15:141
538899139,3cyclictest31368-21tr21:10:141
538899139,3cyclictest15588-21grep23:15:141
538899138,3cyclictest10748-21grep19:20:131
538899137,2cyclictest475-21in:imuxsock19:35:171
538899133,5cyclictest225-21systemd-journal23:36:061
5388991310,2cyclictest28548-21wc22:05:251
538899130,1cyclictest30592-21diskmemload23:20:181
538899130,13cyclictest0-21swapper/123:30:141
538499139,3cyclictest8467-21grep23:10:200
538499139,3cyclictest30441-21grep22:45:140
538499139,3cyclictest14762-21grep22:35:280
538499139,2cyclictest29734-21cut20:00:140
538499139,2cyclictest28085-21grep22:05:190
5384991310,2cyclictest26179-21tr21:00:130
5384991310,2cyclictest20309-21grep20:45:240
5384991310,2cyclictest18548-21wc23:35:260
538499130,2cyclictest0-21swapper/021:25:150
538499130,12cyclictest25549-21cut19:50:160
538499130,0cyclictest0-21swapper/023:17:200
539599129,2cyclictest9443-21grep19:15:213
539599129,2cyclictest6395-21grep19:10:173
539599129,2cyclictest4267-21grep00:25:153
539599129,2cyclictest3955-21wc22:10:193
539599129,2cyclictest39-21ksoftirqd/300:17:573
539599129,2cyclictest20817-21tr00:35:153
539599129,2cyclictest20283-21grep22:00:273
539599129,2cyclictest1840-21wc21:30:243
539599129,2cyclictest14210-21grep19:25:223
539599129,2cyclictest13651-21cut22:35:193
539599126,1cyclictest225-21systemd-journal20:25:003
539599122,3cyclictest225-21systemd-journal19:30:163
5395991210,2cyclictest39-21ksoftirqd/322:42:343
539599120,1cyclictest141rcu_preempt22:45:173
539399129,2cyclictest9619-21cut23:50:142
539399129,2cyclictest6382-21sed19:10:172
539399129,2cyclictest45822sleep221:32:252
539399129,2cyclictest29795-21grep22:45:112
539399129,2cyclictest26469-21ssh00:00:132
539399129,2cyclictest26338-21cron22:05:012
539399129,2cyclictest24466-21head20:55:172
539399129,2cyclictest24201-21grep19:45:272
539399129,2cyclictest21330-21grep20:50:132
539399129,2cyclictest21294-21ssh00:35:172
539399129,2cyclictest13238-21sed20:30:262
539399128,3cyclictest21878-21grep22:40:162
539399128,3cyclictest10332-21ls19:20:002
539399128,1cyclictest0-21swapper/223:00:092
539399124,4cyclictest225-21systemd-journal20:05:002
5393991210,1cyclictest1544-21basename21:50:132
538899129,2cyclictest29744-21wc21:05:231
538899129,2cyclictest24390-21cut20:55:171
538899129,2cyclictest21781-21tail20:50:151
538899129,2cyclictest17654-21wc19:35:001
538899129,2cyclictest11378-21grep21:55:211
538899128,3cyclictest7723-21cron19:15:011
538899128,3cyclictest1756-21cron20:10:001
538899126,2cyclictest225-21systemd-journal20:40:001
5388991210,1cyclictest0-21swapper/123:00:111
538899120,1cyclictest30592-21diskmemload00:37:321
538899120,11cyclictest27357-21dump-pmu-power00:20:001
538499129,3cyclictest13-21ksoftirqd/023:51:410
538499129,2cyclictest7530-21grep20:20:150
538499129,2cyclictest5527-21cat22:30:180
538499129,2cyclictest28361-21wc19:55:180
538499129,2cyclictest25760-21grep23:40:140
538499129,2cyclictest23153-21grep23:00:160
538499129,2cyclictest22285-21grep20:50:200
538499129,2cyclictest20198-21tr22:20:140
538499129,2cyclictest15166-21wc21:20:000
538499128,3cyclictest9856-21cut20:25:170
538499128,2cyclictest27366-21perf00:20:000
538499128,1cyclictest0-21swapper/000:25:010
538499127,2cyclictest225-21systemd-journal21:45:000
538499123,3cyclictest0-21swapper/021:33:550
538499121,7cyclictest5627-21apache_accesses22:50:120
538499120,1cyclictest0-21swapper/000:02:120
539599119,1cyclictest32326-21grep20:05:153
539599118,2cyclictest8775-21expr23:30:153
539599118,2cyclictest5263-21cut22:30:163
539599118,2cyclictest21275-21ntpq19:40:203
539599118,2cyclictest18830-21wget20:45:133
539599117,3cyclictest22483-21wc20:50:243
539599114,6cyclictest17529-21users23:15:263
539599113,4cyclictest225-21systemd-journal20:30:003
539599113,2cyclictest225-21systemd-journal21:40:443
5395991110,1cyclictest39-21ksoftirqd/320:00:003
539599110,10cyclictest21517-21chrt22:20:223
539399118,2cyclictest6154-21taskset20:15:282
539399118,2cyclictest3792-21taskset20:10:442
539399118,2cyclictest29877-21ssh00:20:272
539399118,2cyclictest27638-21cat19:55:142
539399118,2cyclictest27369-21turbostat00:20:002
539399118,2cyclictest26369-21ssh23:20:422
539399118,2cyclictest25543-21expr23:40:142
539399118,2cyclictest20557-21taskset20:46:592
539399118,2cyclictest16798-21chrt21:40:102
539399118,2cyclictest16148-21date20:40:002
539399118,2cyclictest11698-21tail19:20:192
539399114,4cyclictest32-21ksoftirqd/200:25:012
539399113,8cyclictest291irq_work/222:19:282
539399113,6cyclictest10323-21ntp_offset20:25:202
5393991110,1cyclictest0-21swapper/200:31:202
538899119,1cyclictest18297-21taskset20:43:261
538899118,2cyclictest5624-21ssh21:52:041
538899118,2cyclictest3174-21grep00:05:161
538899118,2cyclictest26776-21cut21:00:161
538899118,2cyclictest22775-21tr19:45:141
538899118,2cyclictest2057-21head21:30:251
538899118,2cyclictest19306-21cut20:45:161
538899118,2cyclictest17961-21ssh21:20:401
538899118,2cyclictest14536-21timerandwakeup22:35:271
538899118,2cyclictest11294-21tr00:10:161
538899116,3cyclictest8962-21ssh22:13:361
538899112,8cyclictest3135-21kworker/1:023:10:141
5388991111,0cyclictest0-21swapper/122:21:071
538899110,10cyclictest6081-21sed20:15:251
538899110,10cyclictest25855-21expr23:40:161
538499118,2cyclictest587-21grep21:10:260
538499118,2cyclictest32380-21ssh23:25:100
538499118,2cyclictest3035-21which21:50:220
538499118,2cyclictest30159-21sed21:05:270
538499118,2cyclictest27482sleep000:05:140
538499118,2cyclictest26574-21rm00:38:350
538499118,2cyclictest24781-21ssh22:03:470
538499118,2cyclictest23621-21taskset20:55:090
538499118,2cyclictest23274-21sed19:45:160
538499118,2cyclictest21763-21sed19:40:250
538499118,2cyclictest16727-21ssh22:55:550
538499118,2cyclictest11220-21ssh23:31:040
538499118,2cyclictest10729-21grep21:55:170
538499117,3cyclictest29829-21grep22:25:220
538499110,2cyclictest0-21swapper/020:10:200
538499110,10cyclictest17252-21sed20:40:160
539599108,1cyclictest4772-21ls20:15:143
539599108,1cyclictest30189-21sed20:00:163
539599108,1cyclictest257022sleep320:58:363
539599108,1cyclictest245442sleep319:48:363
539599108,1cyclictest15450-21cat20:35:263
539599108,1cyclictest13353-21chrt20:31:113
539599107,2cyclictest18427-21tune2fs19:35:173
539399108,1cyclictest30244-21taskset21:05:412
539399107,2cyclictest26074-21ntpq19:50:202
539399107,2cyclictest14837-21taskset19:26:532
539399106,3cyclictest8409-21grep20:20:252
538899107,2cyclictest2798-21grep20:10:171
538899105,4cyclictest25062-21cut19:50:141
538899105,4cyclictest23786-21apache220:00:131
538899102,4cyclictest225-21systemd-journal19:45:001
538499108,1cyclictest15747-21expr19:30:140
538499107,2cyclictest8248-21grep19:15:130
538499107,2cyclictest798-21ntpq20:05:210
538499107,2cyclictest2889-21ssh22:10:130
538499107,2cyclictest13235-21grep20:30:260
538499107,2cyclictest10681-21ssh00:10:010
538499101,6cyclictest0-21swapper/020:15:130
53849997,1cyclictest12513-21taskset19:22:290
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional