You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 17:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 30, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26788210982,9sleep30-21swapper/307:08:063
2667129481,9sleep10-21swapper/107:06:341
2669728864,13sleep20-21swapper/207:06:562
2687228269,9sleep00-21swapper/007:09:100
2702099230,22cyclictest0-21swapper/009:39:210
27024992221,1cyclictest0-21swapper/111:49:391
27020992221,1cyclictest0-21swapper/010:16:060
2702099210,6cyclictest0-21swapper/008:30:140
2703099200,19cyclictest0-21swapper/309:51:583
2702799201,1cyclictest19415-21tune2fs08:00:182
2702799201,19cyclictest0-21swapper/211:15:262
27030991916,2cyclictest39-21ksoftirqd/310:15:003
2702799191,16cyclictest0-21swapper/212:29:232
2702499190,19cyclictest0-21swapper/111:27:191
27020991918,1cyclictest0-21swapper/009:24:330
27020991818,0cyclictest0-21swapper/011:00:400
27020991818,0cyclictest0-21swapper/009:42:370
2702099181,10cyclictest4940-21ssh12:25:140
2703099170,2cyclictest0-21swapper/311:42:503
2703099170,2cyclictest0-21swapper/309:14:043
27020991714,2cyclictest31329-21ntp_states10:10:250
2702099171,1cyclictest141rcu_preempt12:20:590
2703099160,6cyclictest0-21swapper/308:50:153
27027991615,1cyclictest30973-21kworker/2:012:10:182
27027991613,3cyclictest37850irq/131-snd_hda10:30:292
2702799161,14cyclictest4552-21ssh12:00:552
2702499169,6cyclictest0-21swapper/109:20:541
2702499163,12cyclictest9883-21kworker/u8:011:52:031
2702099160,14cyclictest0-21swapper/009:27:180
27030991512,3cyclictest39-21ksoftirqd/311:15:303
27027991514,1cyclictest4144-21kworker/2:209:33:192
27027991514,1cyclictest32-21ksoftirqd/211:06:312
2702799151,11cyclictest0-21swapper/210:29:142
2702799150,2cyclictest0-21swapper/211:57:282
27024991512,2cyclictest801-21apt-get08:30:151
2702499150,2cyclictest0-21swapper/109:39:291
2702499150,1cyclictest0-21swapper/109:51:591
2702099150,14cyclictest0-21swapper/009:52:410
58992140,0chrt0-21swapper/110:15:391
2703099147,3cyclictest0-21swapper/312:29:553
2703099144,4cyclictest19647-21diskmemload10:55:223
2703099142,10cyclictest23642-21proc_pri10:30:253
27030991413,1cyclictest0-21swapper/311:47:513
27030991411,2cyclictest30024-21apt-get11:05:143
2703099140,4cyclictest0-21swapper/311:26:583
27027991413,1cyclictest0-21swapper/210:35:192
27027991411,2cyclictest0-21swapper/209:45:362
2702799140,14cyclictest0-21swapper/212:38:352
2702799140,14cyclictest0-21swapper/210:21:462
2702799140,13cyclictest0-21swapper/210:01:302
2702799140,12cyclictest13344-21cut11:45:152
2702799140,12cyclictest12244-21head07:45:182
2702799140,12cyclictest12244-21head07:45:172
2702799140,10cyclictest0-21swapper/208:40:152
2702499141,3cyclictest32484-21ssh11:06:171
27024991410,2cyclictest0-21swapper/109:58:371
2702499140,9cyclictest0-21swapper/110:40:131
2703099137,4cyclictest14800-21cron08:59:593
2703099134,5cyclictest19919-21ssh09:35:263
2703099132,11cyclictest0-21swapper/309:56:343
27030991312,1cyclictest0-21swapper/309:19:363
27030991311,1cyclictest381rcuc/311:10:303
2703099131,10cyclictest0-21swapper/312:35:153
27030991310,2cyclictest39-21ksoftirqd/312:15:003
2702799138,4cyclictest28152-21kworker/2:010:50:362
2702799133,5cyclictest224-21systemd-journal10:10:002
2702799132,3cyclictest13422-21kworker/2:011:30:272
2702799131,7cyclictest0-21swapper/207:40:162
27027991312,1cyclictest7426-21kworker/2:209:52:192
27027991310,2cyclictest26869-21cron11:55:002
27027991310,2cyclictest26826-21head08:15:192
27027991310,2cyclictest21314-21cut11:25:152
27027991310,2cyclictest11939-21grep08:50:272
27027991310,2cyclictest11285-21cut09:30:002
2702499137,6cyclictest20699-21idleruntime-cro11:25:001
2702499133,6cyclictest29646-21munin-run10:10:001
2702499132,7cyclictest32744-21latency_hist11:35:011
2702499132,7cyclictest0-21swapper/109:25:131
27024991310,2cyclictest21930-21grep09:10:271
27024991310,2cyclictest10148-21cut11:15:231
2702499130,2cyclictest0-21swapper/110:25:141
2702499130,12cyclictest18853-21cut12:35:161
2702099139,3cyclictest22305-21wc11:25:200
2702099136,6cyclictest27794-21munin-node09:33:440
27020991310,2cyclictest8676-21ssh11:15:010
27020991310,2cyclictest29192-21grep08:20:180
27020991310,2cyclictest19660-21sleep09:10:000
2703099129,2cyclictest6205-21grep09:25:173
2703099129,2cyclictest4804-21sed08:35:263
2703099129,2cyclictest3960-21grep12:00:273
2703099129,2cyclictest32114-21wc08:25:233
2703099129,2cyclictest2777-21sed07:25:193
2703099129,2cyclictest25829-21head10:05:283
2703099129,2cyclictest24891-21grep11:00:143
2703099129,2cyclictest1910-21tr11:35:273
2703099128,3cyclictest6940-21ssh10:45:043
2703099126,4cyclictest16356-21ssh10:25:163
2703099123,7cyclictest27280-21apt-get11:30:143
2703099122,6cyclictest20126-21apt-get11:50:123
27030991210,1cyclictest5623-21wc07:30:223
2703099120,1cyclictest0-21swapper/310:40:133
2702799129,2cyclictest7970-21grep07:35:242
2702799129,2cyclictest31042-21uniq12:20:162
2702799129,2cyclictest24603-21sed11:00:002
2702799129,2cyclictest18428-21cron08:00:002
2702799129,2cyclictest15623-21sort12:10:002
2702799129,2cyclictest15553-21cut11:20:182
2702799129,2cyclictest12932-21ls12:30:222
2702799128,3cyclictest5904-21ssh10:43:342
2702799128,3cyclictest31683-21cat10:10:282
2702799127,4cyclictest37850irq/131-snd_hda09:22:262
2702799120,11cyclictest27883-21grep07:10:162
2702499129,3cyclictest7483-21grep07:35:181
2702499129,2cyclictest9435-21ssh11:41:451
2702499129,2cyclictest8242-21grep08:45:151
2702499129,2cyclictest4750-21cat08:35:261
2702499129,2cyclictest3787-21tr11:10:231
2702499129,2cyclictest32606-21sed09:45:261
2702499129,2cyclictest32107-21basename07:20:191
2702499129,2cyclictest31600-21tr08:25:181
2702499129,2cyclictest30201-21cut10:10:151
2702499129,2cyclictest23616-21tr12:15:151
2702499129,2cyclictest19670-21sleep09:10:001
2702499129,2cyclictest14352-21sed08:55:261
2702499125,3cyclictest224-21systemd-journal07:34:561
2702499123,5cyclictest23400-21anacron10:04:571
27024991210,1cyclictest26624-21grep11:00:331
2702499120,10cyclictest0-21swapper/112:21:061
2702099129,2cyclictest30722-21sort11:05:290
2702099129,2cyclictest28184-21cron08:20:000
2702099129,2cyclictest21441-21sed11:50:210
2702099129,2cyclictest20855-21grep09:10:170
2702099129,2cyclictest13371-21grep11:45:150
2702099129,2cyclictest12933-21basename10:50:170
2702099129,2cyclictest11361-21grep09:55:170
2702099128,2cyclictest5115-21grep07:30:180
2702099124,6cyclictest20682-21cron11:25:010
2702099120,11cyclictest31562-21grep08:25:180
2703099119,1cyclictest867-21grep08:30:153
2703099118,2cyclictest9987-21ssh10:20:123
2703099118,2cyclictest9948-21cut12:05:193
2703099118,2cyclictest9839-21cut07:40:193
2703099118,2cyclictest9645-21head08:45:273
2703099118,2cyclictest674-21sed09:20:183
2703099118,2cyclictest6316-21tail08:40:193
2703099118,2cyclictest4073-21tr10:15:173
2703099118,2cyclictest29401-21uname08:20:203
2703099118,2cyclictest29175-21grep10:35:193
2703099118,2cyclictest27874-21chrt08:16:083
2703099118,2cyclictest27865-21tr11:55:173
2703099118,2cyclictest26035-21cat09:40:233
2703099118,2cyclictest25519-21ssh12:15:413
2703099118,2cyclictest254542sleep308:13:293
2703099118,2cyclictest23212-21sh08:10:003
2703099118,2cyclictest19467-21grep08:00:183
2703099118,2cyclictest17626-21chrt11:21:213
2703099118,2cyclictest1549-21seq07:22:563
2703099118,2cyclictest1304-21sh12:21:473
2703099118,2cyclictest12963-21tr10:50:163
2703099118,2cyclictest12277-21expr09:30:153
2703099118,2cyclictest11456-21sh12:30:053
2703099112,4cyclictest21884-21sshd10:03:133
2703099110,10cyclictest180692sleep307:55:293
2702799118,2cyclictest6907-21sh10:16:462
2702799118,2cyclictest6529-21sh11:11:522
2702799118,2cyclictest534-21awk08:29:592
2702799118,2cyclictest3185-21ssh11:36:232
2702799118,2cyclictest29777-21ssh09:44:252
2702799118,2cyclictest29712-21grep07:15:162
2702799118,2cyclictest29637-21ntp_states08:20:222
2702799118,2cyclictest2866-21chrt08:34:352
2702799118,2cyclictest28545-21ssh12:18:302
2702799118,2cyclictest27203-21grep09:15:242
2702799118,2cyclictest2541-21df07:25:182
2702799118,2cyclictest24810-21sh11:00:132
2702799118,2cyclictest22207-21ssh09:10:432
2702799118,2cyclictest21767-21tune2fs08:05:172
2702799118,2cyclictest19533-21wc09:35:222
2702799118,2cyclictest1729-21turbostat07:25:002
2702799118,2cyclictest14799-21cron08:59:592
2702799118,2cyclictest11829-21ssh09:55:192
2702799118,2cyclictest10761-21ssh10:48:042
2702799118,2cyclictest10049-21cron08:49:592
2702799113,4cyclictest224-21systemd-journal08:15:012
2702499119,2cyclictest32048-21ssh10:37:501
2702499119,2cyclictest26726-21ssh09:40:501
2702499119,2cyclictest13213-21taskset09:30:231
2702499119,1cyclictest8502-21awk10:45:241
2702499119,1cyclictest3712sleep111:35:051
2702499119,1cyclictest26397-21df09:15:181
2702499119,1cyclictest26184-21sh10:32:561
2702499119,1cyclictest12436-21sh08:55:001
2702499118,2cyclictest9858-21ssh10:20:061
2702499118,2cyclictest9105-21seq07:40:111
2702499118,2cyclictest29395-21grep11:55:271
2702499118,2cyclictest28358-21grep07:10:191
2702499118,2cyclictest24352-21taskset10:59:581
2702499118,2cyclictest23622-21egrep08:10:151
2702499118,2cyclictest2142-21egrep07:25:161
2702499118,2cyclictest16615-21tr07:55:151
2702499118,2cyclictest16368-21cut12:10:171
2702499118,2cyclictest15643-21df_abs09:00:171
2702499118,2cyclictest12749-21ssh10:50:081
2702499118,2cyclictest11987-21expr07:45:161
2702499118,2cyclictest11987-21expr07:45:151
2702499118,2cyclictest11889-21grep12:30:151
2702499116,2cyclictest19647-21diskmemload12:26:581
2702499115,5cyclictest6904-1kworker/u9:212:05:401
2702499115,1cyclictest3397-21sh12:00:221
2702499114,4cyclictest20600-21sshd08:02:131
2702499112,5cyclictest516-21gdbus08:40:241
2702099119,1cyclictest847-21grep07:20:270
2702099118,2cyclictest9446-21grep11:15:180
2702099118,2cyclictest8570-21ssh11:40:430
2702099118,2cyclictest7941-21tail10:45:190
2702099118,2cyclictest5102-21ssh12:01:260
2702099118,2cyclictest4421-21ssh10:41:440
2702099118,2cyclictest3920-21chrt07:27:530
2702099118,2cyclictest31352-21head07:15:280
2702099118,2cyclictest30517-21ssh10:35:430
2702099118,2cyclictest29711-21cat10:10:000
2702099118,2cyclictest2692-21ssh11:35:430
2702099118,2cyclictest26655-21ssh12:16:450
2702099118,2cyclictest26641-21grep09:15:190
2702099118,2cyclictest25040-21cron12:40:000
2702099118,2cyclictest19558-21ssh10:00:350
2702099118,2cyclictest17172-21taskset09:03:550
2702099118,2cyclictest14616-21ssh12:08:550
2702099118,2cyclictest10833-21tail10:20:200
2702099117,3cyclictest16609-21tr07:55:150
2702099117,3cyclictest12666-21wc12:30:200
2702099117,2cyclictest9856-21grep07:40:190
2702099116,5cyclictest17852-21timerwakeupswit12:10:290
2702099110,10cyclictest8741-21chrt07:38:470
2702099110,10cyclictest3339-21ssh09:49:170
188942110,0sleep00-21swapper/010:55:100
2703099108,1cyclictest87192sleep307:38:313
2703099108,1cyclictest14165-21tr07:50:263
2703099108,1cyclictest11470-21chrt07:45:043
2703099108,1cyclictest11470-21chrt07:45:033
2703099107,2cyclictest30833-21sh09:45:143
2703099107,2cyclictest30202-21sed07:15:183
2703099107,2cyclictest27386-21cut07:10:143
2703099107,2cyclictest17487-21chrt09:05:003
2703099103,3cyclictest224-21systemd-journal09:00:003
2702799108,2cyclictest4767-21expr07:30:152
2702799108,1cyclictest6594-21cat11:40:142
2702799108,1cyclictest32892sleep208:35:142
2702799107,2cyclictest18567-21cat09:05:202
2702799107,2cyclictest14011-21taskset07:50:262
2702499108,1cyclictest28785-21cat08:20:161
2702499108,1cyclictest13939-21chrt07:50:191
2702499107,2cyclictest21957-21if_enp0s31f608:05:181
2702499102,7cyclictest31743-21cron07:20:001
2702499100,1cyclictest0-21swapper/108:15:171
2702099108,1cyclictest291132sleep007:11:550
2702099108,1cyclictest27414-21ssh11:55:050
2702099108,1cyclictest25776-21date08:15:000
2702099108,1cyclictest19012-21tr08:00:150
2702099108,1cyclictest18131-21ssh10:25:290
2702099107,2cyclictest6067-21df08:40:160
2702099107,2cyclictest22085-21ssh10:30:080
2702099107,2cyclictest14032-21basename07:50:250
2702099107,2cyclictest11427-21grep08:50:200
2702099107,1cyclictest7913-21chrt08:45:060
2702099103,4cyclictest0-21swapper/011:30:150
2702099101,1cyclictest0-21swapper/008:05:180
2702099100,9cyclictest13388-21chrt07:47:080
2702099100,9cyclictest13388-21chrt07:47:080
270279996,2cyclictest15417-21seq09:00:162
270209997,1cyclictest50482sleep008:37:050
270209991,5cyclictest13503-21smartctl08:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional